[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 13:59:09.961282 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:39137,127.29.194.253:33553,127.29.194.252:44567
I20240501 13:59:09.963022 30475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 13:59:09.964217 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:09.975946 30481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:09.976403 30483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:09.976931 30480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:09.977602 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:09.978510 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:09.978694 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:09.978806 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571949978776 us; error 0 us; skew 500 ppm
I20240501 13:59:09.979166 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:09.982051 30475 webserver.cc:460] Webserver started at http://127.29.194.254:37085/ using document root <none> and password file <none>
I20240501 13:59:09.982878 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:09.982995 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:09.983376 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:09.986891 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "03f6b544be8b480abc5384a781983a8b"
format_stamp: "Formatted at 2024-05-01 13:59:09 on dist-test-slave-kwk7"
I20240501 13:59:09.993039 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20240501 13:59:09.996549 30488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:09.998095 30475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:09.998276 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "03f6b544be8b480abc5384a781983a8b"
format_stamp: "Formatted at 2024-05-01 13:59:09 on dist-test-slave-kwk7"
I20240501 13:59:09.998486 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:10.026757 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.027689 30475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 13:59:10.027993 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.047570 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:39137
I20240501 13:59:10.047591 30539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:39137 every 8 connection(s)
I20240501 13:59:10.049911 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:10.053391 30542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.053715 30540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 13:59:10.053934 30543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.055744 30475 server_base.cc:1034] running on GCE node
W20240501 13:59:10.056252 30545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.056803 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:10.056892 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:10.056975 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571950056937 us; error 0 us; skew 500 ppm
I20240501 13:59:10.057168 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:10.059067 30475 webserver.cc:460] Webserver started at http://127.29.194.253:39885/ using document root <none> and password file <none>
I20240501 13:59:10.059324 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:10.059433 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:10.059581 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:10.060268 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root/instance:
uuid: "4f9d2035052b4752ae3ddb7143f275e6"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.063587 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 13:59:10.065433 30550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.065843 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 13:59:10.065955 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root
uuid: "4f9d2035052b4752ae3ddb7143f275e6"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.066102 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:10.066421 30540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:10.076849 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.077657 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.086907 30540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } has no permanent_uuid. Determining permanent_uuid...
W20240501 13:59:10.088325 30489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.194.253:33553: connect: Connection refused (error 111)
W20240501 13:59:10.090699 30540 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.194.253:33553: Network error: Client connection negotiation failed: client connection to 127.29.194.253:33553: connect: Connection refused (error 111)
I20240501 13:59:10.100840 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.253:33553
I20240501 13:59:10.100898 30604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.253:33553 every 8 connection(s)
I20240501 13:59:10.102344 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 13:59:10.102449 30605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 13:59:10.106173 30607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.106321 30605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } has no permanent_uuid. Determining permanent_uuid...
W20240501 13:59:10.108582 30608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:10.109112 30610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.110808 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:10.111119 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:10.111191 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:10.111281 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571950111228 us; error 0 us; skew 500 ppm
I20240501 13:59:10.111455 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:10.113236 30475 webserver.cc:460] Webserver started at http://127.29.194.252:44247/ using document root <none> and password file <none>
I20240501 13:59:10.113576 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:10.113706 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:10.113876 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:10.114511 30605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:10.114671 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root/instance:
uuid: "1a9d417db3cd45e1a3fb61721a7fe440"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.118263 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 13:59:10.120463 30617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.120891 30475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:10.121076 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root
uuid: "1a9d417db3cd45e1a3fb61721a7fe440"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.121258 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:10.121903 30605 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } has no permanent_uuid. Determining permanent_uuid...
W20240501 13:59:10.124205 30605 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.194.252:44567: Network error: Client connection negotiation failed: client connection to 127.29.194.252:44567: connect: Connection refused (error 111)
I20240501 13:59:10.129382 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.129930 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.140044 30540 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } attempt: 1
I20240501 13:59:10.149784 30540 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:10.149945 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.252:44567
I20240501 13:59:10.150056 30669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.252:44567 every 8 connection(s)
I20240501 13:59:10.151847 30670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:10.155902 30670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:10.157341 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:59:10.163421 30670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:10.165537 30540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: Bootstrap starting.
I20240501 13:59:10.169276 30670 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:10.170190 30540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:10.172199 30540 log.cc:824] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: Log is configured to *not* fsync() on all Append() calls
I20240501 13:59:10.175170 30540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: No bootstrap required, opened a new log
I20240501 13:59:10.176435 30605 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } attempt: 1
I20240501 13:59:10.178432 30670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440: Bootstrap starting.
I20240501 13:59:10.180574 30670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:10.182929 30670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440: No bootstrap required, opened a new log
I20240501 13:59:10.185230 30670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
I20240501 13:59:10.185868 30670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:10.185231 30540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
I20240501 13:59:10.186012 30670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a9d417db3cd45e1a3fb61721a7fe440, State: Initialized, Role: FOLLOWER
I20240501 13:59:10.186079 30540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:10.186187 30540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03f6b544be8b480abc5384a781983a8b, State: Initialized, Role: FOLLOWER
I20240501 13:59:10.186980 30540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
I20240501 13:59:10.186980 30670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
I20240501 13:59:10.188161 30677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.188587 30677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:10.189738 30605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6: Bootstrap starting.
I20240501 13:59:10.189667 30676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.190061 30676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:10.190178 30540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:10.190178 30670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:10.191368 30605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:10.193784 30605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6: No bootstrap required, opened a new log
I20240501 13:59:10.194988 30605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
I20240501 13:59:10.195166 30605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:10.195257 30605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f9d2035052b4752ae3ddb7143f275e6, State: Initialized, Role: FOLLOWER
I20240501 13:59:10.195518 30605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
I20240501 13:59:10.196276 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.196543 30605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:10.196588 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [sys.catalog]: This master's current role is: FOLLOWER
W20240501 13:59:10.208115 30700 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 13:59:10.208364 30700 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 13:59:10.211758 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240501 13:59:10.211890 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20240501 13:59:10.213728 30712 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 13:59:10.213865 30712 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 13:59:10.215008 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:10.215723 30713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 13:59:10.215834 30713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 13:59:10.220242 30714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:10.220670 30715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.222174 30475 server_base.cc:1034] running on GCE node
W20240501 13:59:10.222657 30717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.223065 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:10.223147 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:10.223224 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571950223200 us; error 0 us; skew 500 ppm
I20240501 13:59:10.223419 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:10.224974 30475 webserver.cc:460] Webserver started at http://127.29.194.193:40723/ using document root <none> and password file <none>
I20240501 13:59:10.225232 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:10.225333 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:10.225495 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:10.226245 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "0986c7c4f7644a12afd2cf805bfe596a"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.229128 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 13:59:10.231491 30722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.231936 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.001s
I20240501 13:59:10.232070 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "0986c7c4f7644a12afd2cf805bfe596a"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.232230 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240501 13:59:10.280354 30494 tablet.cc:2369] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 13:59:10.282860 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.283545 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.289211 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:10.292634 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:10.292834 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240501 13:59:10.292976 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:10.293089 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.319146 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:42709
I20240501 13:59:10.319186 30784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:42709 every 8 connection(s)
I20240501 13:59:10.338168 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 13:59:10.344372 30786 heartbeater.cc:347] Connected to a master server at 127.29.194.254:39137
W20240501 13:59:10.347676 30792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.351802 30787 heartbeater.cc:347] Connected to a master server at 127.29.194.253:33553
I20240501 13:59:10.351994 30787 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.353160 30786 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.353240 30785 heartbeater.cc:347] Connected to a master server at 127.29.194.252:44567
I20240501 13:59:10.353427 30785 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.354862 30787 heartbeater.cc:511] Master 127.29.194.253:33553 requested a full tablet report, sending...
I20240501 13:59:10.355867 30786 heartbeater.cc:511] Master 127.29.194.254:39137 requested a full tablet report, sending...
I20240501 13:59:10.356717 30785 heartbeater.cc:511] Master 127.29.194.252:44567 requested a full tablet report, sending...
W20240501 13:59:10.358500 30791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.358883 30569 ts_manager.cc:194] Registered new tserver with Master: 0986c7c4f7644a12afd2cf805bfe596a (127.29.194.193:42709)
I20240501 13:59:10.358912 30505 ts_manager.cc:194] Registered new tserver with Master: 0986c7c4f7644a12afd2cf805bfe596a (127.29.194.193:42709)
I20240501 13:59:10.359428 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:10.359681 30635 ts_manager.cc:194] Registered new tserver with Master: 0986c7c4f7644a12afd2cf805bfe596a (127.29.194.193:42709)
W20240501 13:59:10.360390 30794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.361044 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:10.361124 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:10.361214 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571950361176 us; error 0 us; skew 500 ppm
I20240501 13:59:10.361390 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:10.362990 30475 webserver.cc:460] Webserver started at http://127.29.194.194:38855/ using document root <none> and password file <none>
I20240501 13:59:10.363332 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:10.363488 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:10.363660 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:10.364374 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "cf465764453a4e39a472e1d6d17caf72"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.367373 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 13:59:10.369371 30799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.369763 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 13:59:10.369895 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "cf465764453a4e39a472e1d6d17caf72"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.370035 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:10.382231 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.382992 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.383733 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:10.384830 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:10.384907 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.385030 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:10.385089 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.412138 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:37875
I20240501 13:59:10.412161 30861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:37875 every 8 connection(s)
I20240501 13:59:10.421147 30677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:10.421558 30677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
I20240501 13:59:10.424288 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 13:59:10.430020 30677 leader_election.cc:288] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03f6b544be8b480abc5384a781983a8b (127.29.194.254:39137), 4f9d2035052b4752ae3ddb7143f275e6 (127.29.194.253:33553)
I20240501 13:59:10.429596 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f6b544be8b480abc5384a781983a8b" is_pre_election: true
I20240501 13:59:10.429976 30580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f9d2035052b4752ae3ddb7143f275e6" is_pre_election: true
I20240501 13:59:10.435266 30515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9d417db3cd45e1a3fb61721a7fe440 in term 0.
I20240501 13:59:10.435153 30580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a9d417db3cd45e1a3fb61721a7fe440 in term 0.
I20240501 13:59:10.437899 30618 leader_election.cc:302] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f6b544be8b480abc5384a781983a8b, 1a9d417db3cd45e1a3fb61721a7fe440; no voters:
I20240501 13:59:10.439118 30677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:10.439327 30677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:10.439460 30677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:10.443847 30862 heartbeater.cc:347] Connected to a master server at 127.29.194.252:44567
I20240501 13:59:10.444041 30862 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.444846 30677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
I20240501 13:59:10.446148 30862 heartbeater.cc:511] Master 127.29.194.252:44567 requested a full tablet report, sending...
I20240501 13:59:10.446444 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03f6b544be8b480abc5384a781983a8b"
I20240501 13:59:10.446646 30515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:10.447752 30635 ts_manager.cc:194] Registered new tserver with Master: cf465764453a4e39a472e1d6d17caf72 (127.29.194.194:37875)
I20240501 13:59:10.450441 30580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f9d2035052b4752ae3ddb7143f275e6"
I20240501 13:59:10.450690 30580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:10.451350 30515 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9d417db3cd45e1a3fb61721a7fe440 in term 1.
I20240501 13:59:10.452512 30677 leader_election.cc:288] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [CANDIDATE]: Term 1 election: Requested vote from peers 03f6b544be8b480abc5384a781983a8b (127.29.194.254:39137), 4f9d2035052b4752ae3ddb7143f275e6 (127.29.194.253:33553)
I20240501 13:59:10.454924 30618 leader_election.cc:302] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03f6b544be8b480abc5384a781983a8b, 1a9d417db3cd45e1a3fb61721a7fe440; no voters:
I20240501 13:59:10.456569 30677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:10.457896 30863 heartbeater.cc:347] Connected to a master server at 127.29.194.254:39137
I20240501 13:59:10.458005 30580 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a9d417db3cd45e1a3fb61721a7fe440 in term 1.
I20240501 13:59:10.458710 30863 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.459298 30863 heartbeater.cc:511] Master 127.29.194.254:39137 requested a full tablet report, sending...
I20240501 13:59:10.460868 30505 ts_manager.cc:194] Registered new tserver with Master: cf465764453a4e39a472e1d6d17caf72 (127.29.194.194:37875)
W20240501 13:59:10.462302 30868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.463662 30677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 1 LEADER]: Becoming Leader. State: Replica: 1a9d417db3cd45e1a3fb61721a7fe440, State: Running, Role: LEADER
I20240501 13:59:10.464457 30677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } }
W20240501 13:59:10.467170 30870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.475169 30871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9d417db3cd45e1a3fb61721a7fe440. Latest consensus state: current_term: 1 leader_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.475626 30871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:10.481061 30864 heartbeater.cc:347] Connected to a master server at 127.29.194.253:33553
I20240501 13:59:10.481299 30864 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.481988 30864 heartbeater.cc:511] Master 127.29.194.253:33553 requested a full tablet report, sending...
W20240501 13:59:10.482285 30874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:10.483296 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:10.483494 30569 ts_manager.cc:194] Registered new tserver with Master: cf465764453a4e39a472e1d6d17caf72 (127.29.194.194:37875)
I20240501 13:59:10.483844 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:10.483999 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:10.484079 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571950484051 us; error 0 us; skew 500 ppm
I20240501 13:59:10.484135 30876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:10.484318 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:10.486673 30475 webserver.cc:460] Webserver started at http://127.29.194.195:46205/ using document root <none> and password file <none>
I20240501 13:59:10.486793 30876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:10.486996 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:10.487109 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:10.487284 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:10.488282 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "053cfa3438d44c1c8c79c5795560bee2"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.492506 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240501 13:59:10.495527 30515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 1 FOLLOWER]: Refusing update from remote peer 1a9d417db3cd45e1a3fb61721a7fe440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:10.495527 30580 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 1 FOLLOWER]: Refusing update from remote peer 1a9d417db3cd45e1a3fb61721a7fe440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:10.496585 30871 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:10.497054 30881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.497515 30475 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240501 13:59:10.497524 30677 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:10.497658 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "053cfa3438d44c1c8c79c5795560bee2"
format_stamp: "Formatted at 2024-05-01 13:59:10 on dist-test-slave-kwk7"
I20240501 13:59:10.497804 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:10.502436 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9d417db3cd45e1a3fb61721a7fe440. Latest consensus state: current_term: 1 leader_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.502744 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:10.508881 30884 mvcc.cc:205] Tried to move back new op lower bound from 7022886709218783232 to 7022886709109911552. Current Snapshot: MvccSnapshot[applied={T|T < 7022886709218783232}]
I20240501 13:59:10.510490 30686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.510752 30686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:10.513000 30676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a9d417db3cd45e1a3fb61721a7fe440. Latest consensus state: current_term: 1 leader_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.513379 30676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:10.517853 30876 catalog_manager.cc:1347] Generated new cluster ID: bf627851c9c04ee690886e641b5fae3f
I20240501 13:59:10.517990 30876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:10.519989 30871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.520332 30871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:10.521142 30677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.521576 30677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:10.527141 30676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03f6b544be8b480abc5384a781983a8b" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 39137 } } peers { permanent_uuid: "4f9d2035052b4752ae3ddb7143f275e6" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 33553 } } peers { permanent_uuid: "1a9d417db3cd45e1a3fb61721a7fe440" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 44567 } } }
I20240501 13:59:10.527401 30676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:10.547773 30876 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:10.549672 30876 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:10.556941 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:10.557741 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:10.558570 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:10.559769 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:10.559880 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.559998 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:10.560115 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:10.569164 30876 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440: Generated new TSK 0
I20240501 13:59:10.570418 30876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:10.592955 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:43975
I20240501 13:59:10.592981 30951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:43975 every 8 connection(s)
I20240501 13:59:10.608636 30952 heartbeater.cc:347] Connected to a master server at 127.29.194.252:44567
I20240501 13:59:10.608879 30952 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.609707 30952 heartbeater.cc:511] Master 127.29.194.252:44567 requested a full tablet report, sending...
I20240501 13:59:10.611485 30635 ts_manager.cc:194] Registered new tserver with Master: 053cfa3438d44c1c8c79c5795560bee2 (127.29.194.195:43975)
I20240501 13:59:10.614356 30635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58972
I20240501 13:59:10.617936 30953 heartbeater.cc:347] Connected to a master server at 127.29.194.254:39137
I20240501 13:59:10.618148 30953 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.618664 30953 heartbeater.cc:511] Master 127.29.194.254:39137 requested a full tablet report, sending...
I20240501 13:59:10.620471 30505 ts_manager.cc:194] Registered new tserver with Master: 053cfa3438d44c1c8c79c5795560bee2 (127.29.194.195:43975)
I20240501 13:59:10.621771 30954 heartbeater.cc:347] Connected to a master server at 127.29.194.253:33553
I20240501 13:59:10.621917 30954 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:10.622258 30954 heartbeater.cc:511] Master 127.29.194.253:33553 requested a full tablet report, sending...
I20240501 13:59:10.623347 30569 ts_manager.cc:194] Registered new tserver with Master: 053cfa3438d44c1c8c79c5795560bee2 (127.29.194.195:43975)
I20240501 13:59:10.623819 30475 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007020748s
I20240501 13:59:11.210125 30700 catalog_manager.cc:1259] Loaded cluster ID: bf627851c9c04ee690886e641b5fae3f
I20240501 13:59:11.210233 30700 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6: loading cluster ID for follower catalog manager: success
I20240501 13:59:11.212342 30700 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6: acquiring CA information for follower catalog manager: success
I20240501 13:59:11.213696 30700 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 13:59:11.214891 30712 catalog_manager.cc:1259] Loaded cluster ID: bf627851c9c04ee690886e641b5fae3f
I20240501 13:59:11.214962 30712 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: loading cluster ID for follower catalog manager: success
I20240501 13:59:11.216753 30712 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: acquiring CA information for follower catalog manager: success
I20240501 13:59:11.218122 30712 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 13:59:11.363144 30635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58946
I20240501 13:59:11.450685 30635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58960
I20240501 13:59:11.617921 30952 heartbeater.cc:503] Master 127.29.194.252:44567 was elected leader, sending a full tablet report...
I20240501 13:59:12.211886 30711 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:12.364447 30785 heartbeater.cc:503] Master 127.29.194.252:44567 was elected leader, sending a full tablet report...
I20240501 13:59:12.452279 30862 heartbeater.cc:503] Master 127.29.194.252:44567 was elected leader, sending a full tablet report...
I20240501 13:59:12.686573 30475 test_util.cc:268] Using random seed: -466713130
I20240501 13:59:12.718219 30635 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58984:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 13:59:12.767271 30827 tablet_service.cc:1467] Processing CreateTablet for tablet 072cc17b57dd4e8da769c09180abf8bf (DEFAULT_TABLE table=test-workload [id=b964a767d56c4913810161a77fa32217]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 13:59:12.768368 30827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072cc17b57dd4e8da769c09180abf8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:12.768784 30917 tablet_service.cc:1467] Processing CreateTablet for tablet bb59792676784ee680665bf00c231612 (DEFAULT_TABLE table=test-workload [id=b964a767d56c4913810161a77fa32217]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 13:59:12.769376 30917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb59792676784ee680665bf00c231612. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:12.770306 30750 tablet_service.cc:1467] Processing CreateTablet for tablet cd105efe875b48ce892738bc35b4b568 (DEFAULT_TABLE table=test-workload [id=b964a767d56c4913810161a77fa32217]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 13:59:12.770475 30749 tablet_service.cc:1467] Processing CreateTablet for tablet 9c6b89d142074865882a32c73f4f40f3 (DEFAULT_TABLE table=test-workload [id=b964a767d56c4913810161a77fa32217]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 13:59:12.771546 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd105efe875b48ce892738bc35b4b568. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:12.771842 30749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c6b89d142074865882a32c73f4f40f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:12.783993 30989 tablet_bootstrap.cc:492] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a: Bootstrap starting.
I20240501 13:59:12.784545 30988 tablet_bootstrap.cc:492] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2: Bootstrap starting.
I20240501 13:59:12.785841 30987 tablet_bootstrap.cc:492] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72: Bootstrap starting.
I20240501 13:59:12.786960 30989 tablet_bootstrap.cc:654] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:12.787686 30988 tablet_bootstrap.cc:654] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:12.788333 30987 tablet_bootstrap.cc:654] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:12.790200 30989 tablet_bootstrap.cc:492] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a: No bootstrap required, opened a new log
I20240501 13:59:12.790398 30989 ts_tablet_manager.cc:1367] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 13:59:12.791205 30987 tablet_bootstrap.cc:492] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72: No bootstrap required, opened a new log
I20240501 13:59:12.791409 30987 ts_tablet_manager.cc:1367] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240501 13:59:12.791359 30989 raft_consensus.cc:348] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } }
I20240501 13:59:12.791628 30989 raft_consensus.cc:374] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:12.791775 30989 raft_consensus.cc:729] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0986c7c4f7644a12afd2cf805bfe596a, State: Initialized, Role: FOLLOWER
I20240501 13:59:12.792045 30989 consensus_queue.cc:260] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } }
I20240501 13:59:12.792284 30989 raft_consensus.cc:388] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:12.792400 30989 raft_consensus.cc:482] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:12.792532 30989 raft_consensus.cc:3037] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:12.792508 30987 raft_consensus.cc:348] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf465764453a4e39a472e1d6d17caf72" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 37875 } }
I20240501 13:59:12.792776 30987 raft_consensus.cc:374] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:12.792896 30987 raft_consensus.cc:729] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf465764453a4e39a472e1d6d17caf72, State: Initialized, Role: FOLLOWER
I20240501 13:59:12.793126 30987 consensus_queue.cc:260] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf465764453a4e39a472e1d6d17caf72" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 37875 } }
I20240501 13:59:12.793334 30987 raft_consensus.cc:388] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:12.793437 30987 raft_consensus.cc:482] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:12.793565 30987 raft_consensus.cc:3037] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:12.795874 30989 raft_consensus.cc:504] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } }
I20240501 13:59:12.796139 30989 leader_election.cc:302] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0986c7c4f7644a12afd2cf805bfe596a; no voters:
I20240501 13:59:12.796561 30987 raft_consensus.cc:504] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf465764453a4e39a472e1d6d17caf72" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 37875 } }
I20240501 13:59:12.796947 30987 leader_election.cc:302] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf465764453a4e39a472e1d6d17caf72; no voters:
I20240501 13:59:12.797062 30993 raft_consensus.cc:2781] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:12.797366 30989 leader_election.cc:288] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:12.798213 30988 tablet_bootstrap.cc:492] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2: No bootstrap required, opened a new log
I20240501 13:59:12.798492 30988 ts_tablet_manager.cc:1367] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240501 13:59:12.800372 30988 raft_consensus.cc:348] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053cfa3438d44c1c8c79c5795560bee2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43975 } }
I20240501 13:59:12.800662 30988 raft_consensus.cc:374] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:12.800876 30988 raft_consensus.cc:729] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 053cfa3438d44c1c8c79c5795560bee2, State: Initialized, Role: FOLLOWER
I20240501 13:59:12.801272 30988 consensus_queue.cc:260] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053cfa3438d44c1c8c79c5795560bee2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43975 } }
I20240501 13:59:12.801610 30988 raft_consensus.cc:388] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:12.801769 30988 raft_consensus.cc:482] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:12.801971 30988 raft_consensus.cc:3037] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:12.805776 30988 raft_consensus.cc:504] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053cfa3438d44c1c8c79c5795560bee2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43975 } }
I20240501 13:59:12.806226 30988 leader_election.cc:302] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 053cfa3438d44c1c8c79c5795560bee2; no voters:
I20240501 13:59:12.806354 30989 ts_tablet_manager.cc:1398] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a: Time spent starting tablet: real 0.016s user 0.005s sys 0.010s
I20240501 13:59:12.806957 30987 leader_election.cc:288] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:12.807314 30993 raft_consensus.cc:686] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 LEADER]: Becoming Leader. State: Replica: 0986c7c4f7644a12afd2cf805bfe596a, State: Running, Role: LEADER
I20240501 13:59:12.807642 30994 raft_consensus.cc:2781] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:12.807803 30989 tablet_bootstrap.cc:492] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a: Bootstrap starting.
I20240501 13:59:12.807811 30993 consensus_queue.cc:237] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } }
I20240501 13:59:12.810263 30989 tablet_bootstrap.cc:654] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:12.815038 30997 raft_consensus.cc:2781] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:12.816659 30987 ts_tablet_manager.cc:1398] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72: Time spent starting tablet: real 0.025s user 0.012s sys 0.011s
I20240501 13:59:12.815934 30989 tablet_bootstrap.cc:492] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a: No bootstrap required, opened a new log
I20240501 13:59:12.816964 30989 ts_tablet_manager.cc:1367] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 13:59:12.817297 30994 raft_consensus.cc:686] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 1 LEADER]: Becoming Leader. State: Replica: cf465764453a4e39a472e1d6d17caf72, State: Running, Role: LEADER
I20240501 13:59:12.815287 30988 leader_election.cc:288] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:12.818403 30997 raft_consensus.cc:686] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 1 LEADER]: Becoming Leader. State: Replica: 053cfa3438d44c1c8c79c5795560bee2, State: Running, Role: LEADER
I20240501 13:59:12.819104 30994 consensus_queue.cc:237] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf465764453a4e39a472e1d6d17caf72" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 37875 } }
I20240501 13:59:12.819921 30989 raft_consensus.cc:348] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } }
I20240501 13:59:12.821095 30989 raft_consensus.cc:374] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:12.821262 30989 raft_consensus.cc:729] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0986c7c4f7644a12afd2cf805bfe596a, State: Initialized, Role: FOLLOWER
I20240501 13:59:12.821158 30997 consensus_queue.cc:237] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053cfa3438d44c1c8c79c5795560bee2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43975 } }
I20240501 13:59:12.821794 30989 consensus_queue.cc:260] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } }
I20240501 13:59:12.822108 30989 raft_consensus.cc:388] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:12.822300 30989 raft_consensus.cc:482] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:12.822484 30989 raft_consensus.cc:3037] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:12.822949 30988 ts_tablet_manager.cc:1398] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2: Time spent starting tablet: real 0.024s user 0.009s sys 0.002s
I20240501 13:59:12.826087 30989 raft_consensus.cc:504] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } }
I20240501 13:59:12.826308 30989 leader_election.cc:302] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0986c7c4f7644a12afd2cf805bfe596a; no voters:
I20240501 13:59:12.826498 30989 leader_election.cc:288] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:12.826545 30993 raft_consensus.cc:2781] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:12.826735 30993 raft_consensus.cc:686] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 LEADER]: Becoming Leader. State: Replica: 0986c7c4f7644a12afd2cf805bfe596a, State: Running, Role: LEADER
I20240501 13:59:12.826990 30989 ts_tablet_manager.cc:1398] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a: Time spent starting tablet: real 0.010s user 0.003s sys 0.002s
I20240501 13:59:12.827075 30993 consensus_queue.cc:237] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } }
I20240501 13:59:12.860750 30635 catalog_manager.cc:5484] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0986c7c4f7644a12afd2cf805bfe596a (127.29.194.193). New cstate: current_term: 1 leader_uuid: "0986c7c4f7644a12afd2cf805bfe596a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:12.862048 30634 catalog_manager.cc:5484] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 053cfa3438d44c1c8c79c5795560bee2 (127.29.194.195). New cstate: current_term: 1 leader_uuid: "053cfa3438d44c1c8c79c5795560bee2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "053cfa3438d44c1c8c79c5795560bee2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43975 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:12.866756 30633 catalog_manager.cc:5484] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf465764453a4e39a472e1d6d17caf72 (127.29.194.194). New cstate: current_term: 1 leader_uuid: "cf465764453a4e39a472e1d6d17caf72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf465764453a4e39a472e1d6d17caf72" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 37875 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:12.876168 30633 catalog_manager.cc:5484] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0986c7c4f7644a12afd2cf805bfe596a (127.29.194.193). New cstate: current_term: 1 leader_uuid: "0986c7c4f7644a12afd2cf805bfe596a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0986c7c4f7644a12afd2cf805bfe596a" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42709 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:12.912593 30475 tablet_server.cc:178] TabletServer@127.29.194.193:0 shutting down...
I20240501 13:59:12.923305 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:12.923717 30475 tablet_replica.cc:330] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a: stopping tablet replica
I20240501 13:59:12.924137 30475 raft_consensus.cc:2229] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:12.924430 30475 raft_consensus.cc:2258] T 9c6b89d142074865882a32c73f4f40f3 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:12.925989 30475 tablet_replica.cc:330] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a: stopping tablet replica
I20240501 13:59:12.926141 30475 raft_consensus.cc:2229] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:12.926306 30475 raft_consensus.cc:2258] T cd105efe875b48ce892738bc35b4b568 P 0986c7c4f7644a12afd2cf805bfe596a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:12.932061 30475 tablet_server.cc:194] TabletServer@127.29.194.193:0 shutdown complete.
I20240501 13:59:12.937443 30475 tablet_server.cc:178] TabletServer@127.29.194.194:0 shutting down...
I20240501 13:59:12.946681 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:12.947131 30475 tablet_replica.cc:330] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72: stopping tablet replica
I20240501 13:59:12.947417 30475 raft_consensus.cc:2229] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:12.947611 30475 raft_consensus.cc:2258] T 072cc17b57dd4e8da769c09180abf8bf P cf465764453a4e39a472e1d6d17caf72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:12.962397 30475 tablet_server.cc:194] TabletServer@127.29.194.194:0 shutdown complete.
I20240501 13:59:12.965461 30475 tablet_server.cc:178] TabletServer@127.29.194.195:0 shutting down...
I20240501 13:59:12.974545 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:12.974934 30475 tablet_replica.cc:330] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2: stopping tablet replica
I20240501 13:59:12.975102 30475 raft_consensus.cc:2229] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:12.975266 30475 raft_consensus.cc:2258] T bb59792676784ee680665bf00c231612 P 053cfa3438d44c1c8c79c5795560bee2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:12.979523 30475 tablet_server.cc:194] TabletServer@127.29.194.195:0 shutdown complete.
I20240501 13:59:12.982287 30475 master.cc:537] Master@127.29.194.254:39137 shutting down...
I20240501 13:59:12.990664 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:12.990934 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:12.991056 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 03f6b544be8b480abc5384a781983a8b: stopping tablet replica
I20240501 13:59:13.005898 30475 master.cc:552] Master@127.29.194.254:39137 shutdown complete.
I20240501 13:59:13.011149 30475 master.cc:537] Master@127.29.194.253:33553 shutting down...
I20240501 13:59:13.019155 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:13.019460 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:13.019580 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4f9d2035052b4752ae3ddb7143f275e6: stopping tablet replica
I20240501 13:59:13.034085 30475 master.cc:552] Master@127.29.194.253:33553 shutdown complete.
I20240501 13:59:13.038060 30475 master.cc:537] Master@127.29.194.252:44567 shutting down...
I20240501 13:59:13.045095 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:13.045590 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:13.045749 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1a9d417db3cd45e1a3fb61721a7fe440: stopping tablet replica
I20240501 13:59:13.060972 30475 master.cc:552] Master@127.29.194.252:44567 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3116 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240501 13:59:13.075426 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:43215
I20240501 13:59:13.075938 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:13.079001 31014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.079613 31015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.080134 31017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:13.080678 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:13.081017 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:13.081105 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:13.081167 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571953081148 us; error 0 us; skew 500 ppm
I20240501 13:59:13.081362 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:13.082662 30475 webserver.cc:460] Webserver started at http://127.29.194.254:41009/ using document root <none> and password file <none>
I20240501 13:59:13.082906 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:13.083000 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:13.083156 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:13.083832 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "2d07292c75a94b3f9981568363686c9f"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-kwk7"
I20240501 13:59:13.086416 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 13:59:13.088124 31022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.088549 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20240501 13:59:13.088656 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "2d07292c75a94b3f9981568363686c9f"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-kwk7"
I20240501 13:59:13.088804 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:13.097656 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:13.098246 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:13.112411 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:43215
I20240501 13:59:13.112443 31073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:43215 every 8 connection(s)
I20240501 13:59:13.113896 31074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:13.117969 31074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f: Bootstrap starting.
I20240501 13:59:13.119217 31074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:13.120873 31074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f: No bootstrap required, opened a new log
I20240501 13:59:13.121484 31074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d07292c75a94b3f9981568363686c9f" member_type: VOTER }
I20240501 13:59:13.121619 31074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:13.121680 31074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d07292c75a94b3f9981568363686c9f, State: Initialized, Role: FOLLOWER
I20240501 13:59:13.121930 31074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d07292c75a94b3f9981568363686c9f" member_type: VOTER }
I20240501 13:59:13.122092 31074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:13.122176 31074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:13.122296 31074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:13.124164 31074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d07292c75a94b3f9981568363686c9f" member_type: VOTER }
I20240501 13:59:13.124409 31074 leader_election.cc:302] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d07292c75a94b3f9981568363686c9f; no voters:
I20240501 13:59:13.124856 31074 leader_election.cc:288] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:13.125015 31077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:13.125547 31077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 1 LEADER]: Becoming Leader. State: Replica: 2d07292c75a94b3f9981568363686c9f, State: Running, Role: LEADER
I20240501 13:59:13.125823 31077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d07292c75a94b3f9981568363686c9f" member_type: VOTER }
I20240501 13:59:13.125883 31074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:13.126720 31078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d07292c75a94b3f9981568363686c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d07292c75a94b3f9981568363686c9f" member_type: VOTER } }
I20240501 13:59:13.126798 31079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d07292c75a94b3f9981568363686c9f. Latest consensus state: current_term: 1 leader_uuid: "2d07292c75a94b3f9981568363686c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d07292c75a94b3f9981568363686c9f" member_type: VOTER } }
I20240501 13:59:13.127080 31079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:13.127768 31078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:13.128289 31083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:13.129794 31083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:13.131776 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:59:13.132941 31083 catalog_manager.cc:1347] Generated new cluster ID: 0ee760191b6f41b7aea847a1822afd85
I20240501 13:59:13.133047 31083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:13.147253 31083 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:13.148043 31083 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:13.159523 31083 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f: Generated new TSK 0
I20240501 13:59:13.159849 31083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:13.164453 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:13.168185 31095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.168604 31096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.169135 31098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:13.169721 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:13.170023 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:13.170096 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:13.170192 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571953170146 us; error 0 us; skew 500 ppm
I20240501 13:59:13.170430 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:13.171808 30475 webserver.cc:460] Webserver started at http://127.29.194.193:44383/ using document root <none> and password file <none>
I20240501 13:59:13.172060 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:13.172176 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:13.172330 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:13.173146 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "5f979dcf8ade47c5b1947a66e264addd"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-kwk7"
I20240501 13:59:13.176070 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:13.177839 31103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.178318 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 13:59:13.178439 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "5f979dcf8ade47c5b1947a66e264addd"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-kwk7"
I20240501 13:59:13.178579 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:13.193151 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:13.193771 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:13.194412 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:13.195394 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:13.195466 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.195580 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:13.195636 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.211963 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:42839
I20240501 13:59:13.212009 31165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:42839 every 8 connection(s)
I20240501 13:59:13.213784 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:13.217917 31170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.218375 31171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.219128 31173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:13.220606 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:13.220922 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:13.221020 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:13.221086 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571953221062 us; error 0 us; skew 500 ppm
I20240501 13:59:13.221272 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:13.221515 31166 heartbeater.cc:347] Connected to a master server at 127.29.194.254:43215
I20240501 13:59:13.221653 31166 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:13.222124 31166 heartbeater.cc:511] Master 127.29.194.254:43215 requested a full tablet report, sending...
I20240501 13:59:13.222810 30475 webserver.cc:460] Webserver started at http://127.29.194.194:37339/ using document root <none> and password file <none>
I20240501 13:59:13.223054 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:13.223146 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:13.223122 31039 ts_manager.cc:194] Registered new tserver with Master: 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839)
I20240501 13:59:13.223349 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:13.224009 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "de57af3402884bc5afee432f9f25ee33"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-kwk7"
I20240501 13:59:13.224334 31039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56272
I20240501 13:59:13.226866 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:13.228600 31178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.229012 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 13:59:13.229117 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "de57af3402884bc5afee432f9f25ee33"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-kwk7"
I20240501 13:59:13.229254 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:13.236809 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:13.237365 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:13.237890 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:13.238668 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:13.238746 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.238816 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:13.238869 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.254071 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:38415
I20240501 13:59:13.254104 31240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:38415 every 8 connection(s)
I20240501 13:59:13.255872 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:13.259693 31244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.260654 31245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:13.261379 31247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:13.263250 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:13.263660 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:13.263761 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:13.263835 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571953263795 us; error 0 us; skew 500 ppm
I20240501 13:59:13.264019 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:13.265621 31241 heartbeater.cc:347] Connected to a master server at 127.29.194.254:43215
I20240501 13:59:13.265880 31241 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:13.266122 30475 webserver.cc:460] Webserver started at http://127.29.194.195:35685/ using document root <none> and password file <none>
I20240501 13:59:13.266379 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:13.266484 31241 heartbeater.cc:511] Master 127.29.194.254:43215 requested a full tablet report, sending...
I20240501 13:59:13.266508 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:13.266685 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:13.267463 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "7266f74949944f78bb3f0dd836af30ae"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-kwk7"
I20240501 13:59:13.267509 31039 ts_manager.cc:194] Registered new tserver with Master: de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:13.268281 31039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56280
I20240501 13:59:13.270164 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 13:59:13.271798 31252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.272132 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 13:59:13.272248 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "7266f74949944f78bb3f0dd836af30ae"
format_stamp: "Formatted at 2024-05-01 13:59:13 on dist-test-slave-kwk7"
I20240501 13:59:13.272397 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:13.286324 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:13.286960 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:13.287554 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:13.288452 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:13.288525 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.288647 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:13.288733 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:13.305619 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:41005
I20240501 13:59:13.305648 31314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:41005 every 8 connection(s)
I20240501 13:59:13.312637 31315 heartbeater.cc:347] Connected to a master server at 127.29.194.254:43215
I20240501 13:59:13.313369 31315 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:13.313812 31315 heartbeater.cc:511] Master 127.29.194.254:43215 requested a full tablet report, sending...
I20240501 13:59:13.314826 31039 ts_manager.cc:194] Registered new tserver with Master: 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:13.315667 30475 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.009073572s
I20240501 13:59:13.315884 31039 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56286
I20240501 13:59:14.225757 31166 heartbeater.cc:503] Master 127.29.194.254:43215 was elected leader, sending a full tablet report...
I20240501 13:59:14.269567 31241 heartbeater.cc:503] Master 127.29.194.254:43215 was elected leader, sending a full tablet report...
I20240501 13:59:14.317365 31315 heartbeater.cc:503] Master 127.29.194.254:43215 was elected leader, sending a full tablet report...
I20240501 13:59:14.343989 30475 test_util.cc:268] Using random seed: -465055716
I20240501 13:59:14.356880 31039 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56288:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 13:59:14.357599 31039 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 13:59:14.392496 31131 tablet_service.cc:1467] Processing CreateTablet for tablet 71d9917dafba40d0a62fc11f2a03ff13 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 13:59:14.392580 31130 tablet_service.cc:1467] Processing CreateTablet for tablet 73d5a368fc6343c894d83d60e87208b2 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 13:59:14.393216 31131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d9917dafba40d0a62fc11f2a03ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.393143 31129 tablet_service.cc:1467] Processing CreateTablet for tablet b21b80f7c34f4e11a7b352f576e8e744 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 13:59:14.393366 31126 tablet_service.cc:1467] Processing CreateTablet for tablet e5c00b157eab43428141fe8508ac9bec (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 13:59:14.394843 31127 tablet_service.cc:1467] Processing CreateTablet for tablet f3db8b744ada4840b5a8237a813211fb (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 13:59:14.395117 31128 tablet_service.cc:1467] Processing CreateTablet for tablet f0a6fcec53d245a4ac0c66e278405a59 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 13:59:14.396271 31124 tablet_service.cc:1467] Processing CreateTablet for tablet de38b193e71042be81354e5fab815100 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 13:59:14.396791 31125 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce08af8a10f439c97841daa52a9185c (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 13:59:14.397281 31129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b80f7c34f4e11a7b352f576e8e744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.397540 31126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c00b157eab43428141fe8508ac9bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.397748 31128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a6fcec53d245a4ac0c66e278405a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.400873 31130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d5a368fc6343c894d83d60e87208b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.401144 31125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce08af8a10f439c97841daa52a9185c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.403456 31280 tablet_service.cc:1467] Processing CreateTablet for tablet 71d9917dafba40d0a62fc11f2a03ff13 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 13:59:14.403981 31280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d9917dafba40d0a62fc11f2a03ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.405066 31124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de38b193e71042be81354e5fab815100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.409049 31202 tablet_service.cc:1467] Processing CreateTablet for tablet f3db8b744ada4840b5a8237a813211fb (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 13:59:14.409080 31205 tablet_service.cc:1467] Processing CreateTablet for tablet 73d5a368fc6343c894d83d60e87208b2 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 13:59:14.409746 31202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3db8b744ada4840b5a8237a813211fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.409637 31204 tablet_service.cc:1467] Processing CreateTablet for tablet b21b80f7c34f4e11a7b352f576e8e744 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 13:59:14.409969 31205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d5a368fc6343c894d83d60e87208b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.410248 31204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b80f7c34f4e11a7b352f576e8e744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.410641 31201 tablet_service.cc:1467] Processing CreateTablet for tablet e5c00b157eab43428141fe8508ac9bec (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 13:59:14.411224 31201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c00b157eab43428141fe8508ac9bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.412907 31203 tablet_service.cc:1467] Processing CreateTablet for tablet f0a6fcec53d245a4ac0c66e278405a59 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 13:59:14.413084 31200 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce08af8a10f439c97841daa52a9185c (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 13:59:14.413417 31203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a6fcec53d245a4ac0c66e278405a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.413813 31199 tablet_service.cc:1467] Processing CreateTablet for tablet de38b193e71042be81354e5fab815100 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 13:59:14.414803 31199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de38b193e71042be81354e5fab815100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.416874 31200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce08af8a10f439c97841daa52a9185c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.408499 31206 tablet_service.cc:1467] Processing CreateTablet for tablet 71d9917dafba40d0a62fc11f2a03ff13 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 13:59:14.417456 31206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d9917dafba40d0a62fc11f2a03ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.421078 31127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3db8b744ada4840b5a8237a813211fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.431083 31278 tablet_service.cc:1467] Processing CreateTablet for tablet b21b80f7c34f4e11a7b352f576e8e744 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 13:59:14.431931 31279 tablet_service.cc:1467] Processing CreateTablet for tablet 73d5a368fc6343c894d83d60e87208b2 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 13:59:14.435750 31276 tablet_service.cc:1467] Processing CreateTablet for tablet f3db8b744ada4840b5a8237a813211fb (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 13:59:14.438097 31277 tablet_service.cc:1467] Processing CreateTablet for tablet f0a6fcec53d245a4ac0c66e278405a59 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 13:59:14.440361 31273 tablet_service.cc:1467] Processing CreateTablet for tablet de38b193e71042be81354e5fab815100 (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 13:59:14.440726 31278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b21b80f7c34f4e11a7b352f576e8e744. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.440972 31274 tablet_service.cc:1467] Processing CreateTablet for tablet 3ce08af8a10f439c97841daa52a9185c (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 13:59:14.441190 31275 tablet_service.cc:1467] Processing CreateTablet for tablet e5c00b157eab43428141fe8508ac9bec (DEFAULT_TABLE table=test-workload [id=9851ada3747947fa87f8e173d00a1390]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 13:59:14.444507 31276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3db8b744ada4840b5a8237a813211fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.445722 31274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ce08af8a10f439c97841daa52a9185c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.445955 31277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0a6fcec53d245a4ac0c66e278405a59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.446729 31273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de38b193e71042be81354e5fab815100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.448463 31279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73d5a368fc6343c894d83d60e87208b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.449719 31275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c00b157eab43428141fe8508ac9bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:14.453161 31336 tablet_bootstrap.cc:492] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33: Bootstrap starting.
I20240501 13:59:14.456429 31335 tablet_bootstrap.cc:492] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd: Bootstrap starting.
I20240501 13:59:14.476300 31336 tablet_bootstrap.cc:654] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.476800 31335 tablet_bootstrap.cc:654] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.481793 31337 tablet_bootstrap.cc:492] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae: Bootstrap starting.
I20240501 13:59:14.484882 31337 tablet_bootstrap.cc:654] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.486788 31336 tablet_bootstrap.cc:492] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33: No bootstrap required, opened a new log
I20240501 13:59:14.487007 31336 ts_tablet_manager.cc:1367] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.002s
I20240501 13:59:14.488082 31336 raft_consensus.cc:348] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.488384 31336 raft_consensus.cc:374] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.488512 31336 raft_consensus.cc:729] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.489012 31336 consensus_queue.cc:260] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.490142 31336 ts_tablet_manager.cc:1398] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 13:59:14.490499 31336 tablet_bootstrap.cc:492] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33: Bootstrap starting.
I20240501 13:59:14.491530 31337 tablet_bootstrap.cc:492] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae: No bootstrap required, opened a new log
I20240501 13:59:14.491675 31337 ts_tablet_manager.cc:1367] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20240501 13:59:14.492383 31336 tablet_bootstrap.cc:654] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.492449 31337 raft_consensus.cc:348] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.492645 31337 raft_consensus.cc:374] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.492761 31335 tablet_bootstrap.cc:492] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd: No bootstrap required, opened a new log
I20240501 13:59:14.492792 31337 raft_consensus.cc:729] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.492931 31335 ts_tablet_manager.cc:1367] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.008s
I20240501 13:59:14.493638 31337 consensus_queue.cc:260] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.494453 31335 raft_consensus.cc:348] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.494725 31335 raft_consensus.cc:374] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.494868 31335 raft_consensus.cc:729] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.495225 31335 consensus_queue.cc:260] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.496309 31335 ts_tablet_manager.cc:1398] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 13:59:14.496721 31335 tablet_bootstrap.cc:492] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd: Bootstrap starting.
I20240501 13:59:14.499123 31335 tablet_bootstrap.cc:654] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.501713 31337 ts_tablet_manager.cc:1398] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae: Time spent starting tablet: real 0.009s user 0.004s sys 0.005s
I20240501 13:59:14.502278 31337 tablet_bootstrap.cc:492] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae: Bootstrap starting.
I20240501 13:59:14.503883 31335 tablet_bootstrap.cc:492] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd: No bootstrap required, opened a new log
I20240501 13:59:14.504082 31335 ts_tablet_manager.cc:1367] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240501 13:59:14.504175 31337 tablet_bootstrap.cc:654] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.504488 31336 tablet_bootstrap.cc:492] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33: No bootstrap required, opened a new log
I20240501 13:59:14.504675 31336 ts_tablet_manager.cc:1367] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240501 13:59:14.505975 31335 raft_consensus.cc:348] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.506383 31335 raft_consensus.cc:374] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.506497 31335 raft_consensus.cc:729] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.506603 31336 raft_consensus.cc:348] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.506855 31336 raft_consensus.cc:374] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.506973 31336 raft_consensus.cc:729] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.506829 31335 consensus_queue.cc:260] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.507244 31336 consensus_queue.cc:260] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.507586 31335 ts_tablet_manager.cc:1398] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:14.507736 31336 ts_tablet_manager.cc:1398] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 13:59:14.507967 31335 tablet_bootstrap.cc:492] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd: Bootstrap starting.
I20240501 13:59:14.509474 31336 tablet_bootstrap.cc:492] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33: Bootstrap starting.
I20240501 13:59:14.510191 31335 tablet_bootstrap.cc:654] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.511324 31336 tablet_bootstrap.cc:654] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.514827 31336 tablet_bootstrap.cc:492] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33: No bootstrap required, opened a new log
I20240501 13:59:14.514883 31337 tablet_bootstrap.cc:492] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae: No bootstrap required, opened a new log
I20240501 13:59:14.514897 31335 tablet_bootstrap.cc:492] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd: No bootstrap required, opened a new log
I20240501 13:59:14.514995 31336 ts_tablet_manager.cc:1367] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240501 13:59:14.515097 31337 ts_tablet_manager.cc:1367] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240501 13:59:14.515095 31335 ts_tablet_manager.cc:1367] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 13:59:14.515686 31336 raft_consensus.cc:348] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.515890 31336 raft_consensus.cc:374] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.516012 31336 raft_consensus.cc:729] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.515982 31335 raft_consensus.cc:348] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.515901 31337 raft_consensus.cc:348] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.516176 31335 raft_consensus.cc:374] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.516212 31337 raft_consensus.cc:374] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.516301 31337 raft_consensus.cc:729] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.516310 31335 raft_consensus.cc:729] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.516307 31336 consensus_queue.cc:260] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.516623 31337 consensus_queue.cc:260] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.516651 31335 consensus_queue.cc:260] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.516996 31336 ts_tablet_manager.cc:1398] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:14.517207 31335 ts_tablet_manager.cc:1398] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 13:59:14.517262 31336 tablet_bootstrap.cc:492] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33: Bootstrap starting.
I20240501 13:59:14.517486 31335 tablet_bootstrap.cc:492] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd: Bootstrap starting.
I20240501 13:59:14.517781 31337 ts_tablet_manager.cc:1398] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 13:59:14.519672 31337 tablet_bootstrap.cc:492] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae: Bootstrap starting.
I20240501 13:59:14.519713 31336 tablet_bootstrap.cc:654] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.520843 31335 tablet_bootstrap.cc:654] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.521647 31337 tablet_bootstrap.cc:654] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.528965 31336 tablet_bootstrap.cc:492] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33: No bootstrap required, opened a new log
I20240501 13:59:14.529228 31336 ts_tablet_manager.cc:1367] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240501 13:59:14.530542 31335 tablet_bootstrap.cc:492] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd: No bootstrap required, opened a new log
I20240501 13:59:14.530735 31335 ts_tablet_manager.cc:1367] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240501 13:59:14.530833 31336 raft_consensus.cc:348] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.531067 31337 tablet_bootstrap.cc:492] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae: No bootstrap required, opened a new log
I20240501 13:59:14.531091 31336 raft_consensus.cc:374] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.531209 31336 raft_consensus.cc:729] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.531248 31337 ts_tablet_manager.cc:1367] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240501 13:59:14.531548 31335 raft_consensus.cc:348] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.531549 31336 consensus_queue.cc:260] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.531803 31335 raft_consensus.cc:374] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.531893 31335 raft_consensus.cc:729] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.531961 31337 raft_consensus.cc:348] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.532157 31337 raft_consensus.cc:374] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.532150 31335 consensus_queue.cc:260] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.532325 31336 ts_tablet_manager.cc:1398] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:14.532295 31337 raft_consensus.cc:729] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.532608 31336 tablet_bootstrap.cc:492] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33: Bootstrap starting.
I20240501 13:59:14.532709 31335 ts_tablet_manager.cc:1398] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:14.532933 31337 consensus_queue.cc:260] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.533505 31337 ts_tablet_manager.cc:1398] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:14.533991 31344 raft_consensus.cc:482] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.534128 31344 raft_consensus.cc:504] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.534600 31336 tablet_bootstrap.cc:654] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.535275 31344 leader_election.cc:288] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.535432 31337 tablet_bootstrap.cc:492] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae: Bootstrap starting.
I20240501 13:59:14.536336 31335 tablet_bootstrap.cc:492] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd: Bootstrap starting.
I20240501 13:59:14.537297 31337 tablet_bootstrap.cc:654] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.538625 31335 tablet_bootstrap.cc:654] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.549240 31336 tablet_bootstrap.cc:492] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33: No bootstrap required, opened a new log
I20240501 13:59:14.549448 31336 ts_tablet_manager.cc:1367] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240501 13:59:14.550580 31336 raft_consensus.cc:348] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.550849 31336 raft_consensus.cc:374] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.550959 31336 raft_consensus.cc:729] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.551232 31336 consensus_queue.cc:260] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.551841 31336 ts_tablet_manager.cc:1398] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 13:59:14.552160 31336 tablet_bootstrap.cc:492] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33: Bootstrap starting.
I20240501 13:59:14.554351 31336 tablet_bootstrap.cc:654] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.557510 31335 tablet_bootstrap.cc:492] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd: No bootstrap required, opened a new log
I20240501 13:59:14.557700 31335 ts_tablet_manager.cc:1367] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20240501 13:59:14.558197 31290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c00b157eab43428141fe8508ac9bec" candidate_uuid: "5f979dcf8ade47c5b1947a66e264addd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae" is_pre_election: true
I20240501 13:59:14.558545 31290 raft_consensus.cc:2454] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f979dcf8ade47c5b1947a66e264addd in term 0.
I20240501 13:59:14.558619 31335 raft_consensus.cc:348] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.558827 31335 raft_consensus.cc:374] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.558952 31335 raft_consensus.cc:729] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.559103 31104 leader_election.cc:302] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters:
I20240501 13:59:14.559248 31335 consensus_queue.cc:260] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.559401 31344 raft_consensus.cc:2781] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.559500 31344 raft_consensus.cc:482] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.559605 31344 raft_consensus.cc:3037] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.562286 31336 tablet_bootstrap.cc:492] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33: No bootstrap required, opened a new log
I20240501 13:59:14.562532 31336 ts_tablet_manager.cc:1367] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 13:59:14.562711 31344 raft_consensus.cc:504] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.562906 31216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c00b157eab43428141fe8508ac9bec" candidate_uuid: "5f979dcf8ade47c5b1947a66e264addd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33" is_pre_election: true
I20240501 13:59:14.563589 31336 raft_consensus.cc:348] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.563889 31336 raft_consensus.cc:374] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240501 13:59:14.563994 31106 leader_election.cc:341] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:14.564610 31216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c00b157eab43428141fe8508ac9bec" candidate_uuid: "5f979dcf8ade47c5b1947a66e264addd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33"
I20240501 13:59:14.563995 31336 raft_consensus.cc:729] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Initialized, Role: FOLLOWER
W20240501 13:59:14.565137 31106 leader_election.cc:341] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415): Illegal state: must be running to vote when last-logged opid is not known
I20240501 13:59:14.565083 31336 consensus_queue.cc:260] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.565524 31344 leader_election.cc:288] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 election: Requested vote from peers de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.565963 31337 tablet_bootstrap.cc:492] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae: No bootstrap required, opened a new log
I20240501 13:59:14.566251 31335 ts_tablet_manager.cc:1398] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20240501 13:59:14.566252 31337 ts_tablet_manager.cc:1367] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.011s
I20240501 13:59:14.566593 31335 tablet_bootstrap.cc:492] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd: Bootstrap starting.
I20240501 13:59:14.566797 31290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c00b157eab43428141fe8508ac9bec" candidate_uuid: "5f979dcf8ade47c5b1947a66e264addd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae"
I20240501 13:59:14.566993 31290 raft_consensus.cc:3037] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.568122 31337 raft_consensus.cc:348] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.568270 31336 ts_tablet_manager.cc:1398] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240501 13:59:14.568588 31337 raft_consensus.cc:374] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.569041 31336 tablet_bootstrap.cc:492] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33: Bootstrap starting.
I20240501 13:59:14.569051 31337 raft_consensus.cc:729] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.570345 31337 consensus_queue.cc:260] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.571504 31337 ts_tablet_manager.cc:1398] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:14.573194 31337 tablet_bootstrap.cc:492] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae: Bootstrap starting.
I20240501 13:59:14.573233 31290 raft_consensus.cc:2454] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f979dcf8ade47c5b1947a66e264addd in term 1.
I20240501 13:59:14.573711 31104 leader_election.cc:302] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters: de57af3402884bc5afee432f9f25ee33
I20240501 13:59:14.574977 31344 raft_consensus.cc:2781] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.575145 31344 raft_consensus.cc:686] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 1 LEADER]: Becoming Leader. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Running, Role: LEADER
I20240501 13:59:14.575642 31344 consensus_queue.cc:237] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.576836 31336 tablet_bootstrap.cc:654] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.580565 31337 tablet_bootstrap.cc:654] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.582953 31038 catalog_manager.cc:5484] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193). New cstate: current_term: 1 leader_uuid: "5f979dcf8ade47c5b1947a66e264addd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:14.584820 31335 tablet_bootstrap.cc:654] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.590612 31335 tablet_bootstrap.cc:492] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd: No bootstrap required, opened a new log
I20240501 13:59:14.590844 31335 ts_tablet_manager.cc:1367] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20240501 13:59:14.590983 31337 tablet_bootstrap.cc:492] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae: No bootstrap required, opened a new log
I20240501 13:59:14.591061 31336 tablet_bootstrap.cc:492] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33: No bootstrap required, opened a new log
I20240501 13:59:14.591184 31336 ts_tablet_manager.cc:1367] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.008s
I20240501 13:59:14.591182 31337 ts_tablet_manager.cc:1367] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20240501 13:59:14.591734 31335 raft_consensus.cc:348] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.591951 31335 raft_consensus.cc:374] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.592065 31335 raft_consensus.cc:729] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.592072 31336 raft_consensus.cc:348] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.592124 31337 raft_consensus.cc:348] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.592358 31336 raft_consensus.cc:374] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.592383 31337 raft_consensus.cc:374] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.592490 31336 raft_consensus.cc:729] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.592519 31337 raft_consensus.cc:729] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.592456 31335 consensus_queue.cc:260] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.592793 31336 consensus_queue.cc:260] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.592794 31337 consensus_queue.cc:260] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.593089 31335 ts_tablet_manager.cc:1398] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:14.593361 31336 ts_tablet_manager.cc:1398] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:14.593500 31335 tablet_bootstrap.cc:492] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd: Bootstrap starting.
I20240501 13:59:14.593678 31336 tablet_bootstrap.cc:492] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33: Bootstrap starting.
I20240501 13:59:14.595121 31337 ts_tablet_manager.cc:1398] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 13:59:14.595395 31337 tablet_bootstrap.cc:492] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae: Bootstrap starting.
I20240501 13:59:14.595562 31335 tablet_bootstrap.cc:654] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.597648 31337 tablet_bootstrap.cc:654] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.601747 31336 tablet_bootstrap.cc:654] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.603296 31335 tablet_bootstrap.cc:492] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd: No bootstrap required, opened a new log
I20240501 13:59:14.603513 31335 ts_tablet_manager.cc:1367] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 13:59:14.604491 31336 tablet_bootstrap.cc:492] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33: No bootstrap required, opened a new log
I20240501 13:59:14.604647 31336 ts_tablet_manager.cc:1367] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240501 13:59:14.604497 31335 raft_consensus.cc:348] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.604676 31337 tablet_bootstrap.cc:492] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae: No bootstrap required, opened a new log
I20240501 13:59:14.604861 31335 raft_consensus.cc:374] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.604976 31337 ts_tablet_manager.cc:1367] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20240501 13:59:14.605028 31335 raft_consensus.cc:729] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.605381 31335 consensus_queue.cc:260] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.605455 31336 raft_consensus.cc:348] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.605683 31336 raft_consensus.cc:374] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.605755 31336 raft_consensus.cc:729] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.605636 31337 raft_consensus.cc:348] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.605938 31335 ts_tablet_manager.cc:1398] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:14.606015 31337 raft_consensus.cc:374] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.606107 31337 raft_consensus.cc:729] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.606155 31336 consensus_queue.cc:260] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.606361 31335 tablet_bootstrap.cc:492] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd: Bootstrap starting.
I20240501 13:59:14.606700 31336 ts_tablet_manager.cc:1398] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 13:59:14.606971 31337 consensus_queue.cc:260] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.608007 31337 ts_tablet_manager.cc:1398] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 13:59:14.608119 31335 tablet_bootstrap.cc:654] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.608246 31337 tablet_bootstrap.cc:492] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae: Bootstrap starting.
I20240501 13:59:14.610486 31337 tablet_bootstrap.cc:654] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.612636 31335 tablet_bootstrap.cc:492] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd: No bootstrap required, opened a new log
I20240501 13:59:14.612845 31335 ts_tablet_manager.cc:1367] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 13:59:14.613881 31337 tablet_bootstrap.cc:492] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae: No bootstrap required, opened a new log
I20240501 13:59:14.613759 31335 raft_consensus.cc:348] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.614048 31335 raft_consensus.cc:374] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.614066 31337 ts_tablet_manager.cc:1367] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240501 13:59:14.614164 31335 raft_consensus.cc:729] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.614421 31335 consensus_queue.cc:260] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.614684 31337 raft_consensus.cc:348] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.614852 31337 raft_consensus.cc:374] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.614894 31335 ts_tablet_manager.cc:1398] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 13:59:14.614925 31337 raft_consensus.cc:729] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.615090 31337 consensus_queue.cc:260] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.615542 31337 ts_tablet_manager.cc:1398] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 13:59:14.615927 31337 tablet_bootstrap.cc:492] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae: Bootstrap starting.
I20240501 13:59:14.617539 31337 tablet_bootstrap.cc:654] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:14.619599 31337 tablet_bootstrap.cc:492] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae: No bootstrap required, opened a new log
I20240501 13:59:14.619751 31337 ts_tablet_manager.cc:1367] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 13:59:14.620849 31337 raft_consensus.cc:348] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.621013 31337 raft_consensus.cc:374] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:14.621068 31337 raft_consensus.cc:729] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Initialized, Role: FOLLOWER
I20240501 13:59:14.621275 31337 consensus_queue.cc:260] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.621693 31337 ts_tablet_manager.cc:1398] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:14.634063 31342 raft_consensus.cc:482] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.634264 31342 raft_consensus.cc:504] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.634629 31341 raft_consensus.cc:482] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.634779 31341 raft_consensus.cc:504] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.635567 31342 leader_election.cc:288] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:14.635804 31341 leader_election.cc:288] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.648414 31341 raft_consensus.cc:482] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.648914 31341 raft_consensus.cc:504] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.650043 31341 leader_election.cc:288] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.651314 31342 raft_consensus.cc:482] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.651648 31342 raft_consensus.cc:504] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.652763 31342 leader_election.cc:288] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:14.653215 31342 raft_consensus.cc:482] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.653424 31342 raft_consensus.cc:504] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.654858 31342 leader_election.cc:288] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:14.660861 31139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce08af8a10f439c97841daa52a9185c" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd" is_pre_election: true
I20240501 13:59:14.661422 31139 raft_consensus.cc:2454] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 0.
I20240501 13:59:14.660477 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d9917dafba40d0a62fc11f2a03ff13" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd" is_pre_election: true
I20240501 13:59:14.663357 31256 leader_election.cc:302] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters:
I20240501 13:59:14.663357 31141 raft_consensus.cc:2454] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 0.
I20240501 13:59:14.663981 31374 raft_consensus.cc:2781] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.663977 31256 leader_election.cc:302] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters:
I20240501 13:59:14.664135 31374 raft_consensus.cc:482] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.662343 31140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de38b193e71042be81354e5fab815100" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd" is_pre_election: true
I20240501 13:59:14.664678 31342 raft_consensus.cc:2781] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.664955 31374 raft_consensus.cc:3037] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.665141 31216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d9917dafba40d0a62fc11f2a03ff13" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33" is_pre_election: true
I20240501 13:59:14.665453 31290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d5a368fc6343c894d83d60e87208b2" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae" is_pre_election: true
I20240501 13:59:14.665587 31140 raft_consensus.cc:2454] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 0.
I20240501 13:59:14.665673 31289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce08af8a10f439c97841daa52a9185c" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae" is_pre_election: true
I20240501 13:59:14.665500 31215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de38b193e71042be81354e5fab815100" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33" is_pre_election: true
I20240501 13:59:14.666085 31342 raft_consensus.cc:482] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.666168 31216 raft_consensus.cc:2454] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 0.
I20240501 13:59:14.666198 31342 raft_consensus.cc:3037] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.665941 31290 raft_consensus.cc:2454] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 0.
I20240501 13:59:14.666052 31256 leader_election.cc:302] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters:
I20240501 13:59:14.666741 31215 raft_consensus.cc:2454] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 0.
I20240501 13:59:14.666412 31214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce08af8a10f439c97841daa52a9185c" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33" is_pre_election: true
I20240501 13:59:14.667271 31179 leader_election.cc:302] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7266f74949944f78bb3f0dd836af30ae, de57af3402884bc5afee432f9f25ee33; no voters:
I20240501 13:59:14.667388 31214 raft_consensus.cc:2454] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 0.
I20240501 13:59:14.667588 31341 raft_consensus.cc:2781] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.667682 31341 raft_consensus.cc:482] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.667788 31341 raft_consensus.cc:3037] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.669929 31140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d5a368fc6343c894d83d60e87208b2" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd" is_pre_election: true
I20240501 13:59:14.670116 31140 raft_consensus.cc:2454] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 0.
I20240501 13:59:14.670398 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce08af8a10f439c97841daa52a9185c" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd" is_pre_election: true
I20240501 13:59:14.670586 31141 raft_consensus.cc:2454] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 0.
I20240501 13:59:14.670717 31341 raft_consensus.cc:504] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.671190 31182 leader_election.cc:302] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, de57af3402884bc5afee432f9f25ee33; no voters:
I20240501 13:59:14.671468 31342 raft_consensus.cc:504] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.671936 31290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d5a368fc6343c894d83d60e87208b2" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae"
I20240501 13:59:14.672966 31290 raft_consensus.cc:3037] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.673373 31342 leader_election.cc:288] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 election: Requested vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:14.673704 31342 raft_consensus.cc:2781] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.673787 31214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d9917dafba40d0a62fc11f2a03ff13" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33"
I20240501 13:59:14.673894 31342 raft_consensus.cc:482] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.673995 31214 raft_consensus.cc:3037] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.674219 31342 raft_consensus.cc:3037] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.676856 31341 leader_election.cc:288] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 election: Requested vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.677084 31341 raft_consensus.cc:2781] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.677141 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d9917dafba40d0a62fc11f2a03ff13" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd"
I20240501 13:59:14.677353 31141 raft_consensus.cc:3037] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.677460 31290 raft_consensus.cc:2454] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 1.
I20240501 13:59:14.677220 31341 raft_consensus.cc:482] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.677850 31341 raft_consensus.cc:3037] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.677986 31179 leader_election.cc:302] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7266f74949944f78bb3f0dd836af30ae, de57af3402884bc5afee432f9f25ee33; no voters:
I20240501 13:59:14.678017 31342 raft_consensus.cc:504] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.680632 31215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de38b193e71042be81354e5fab815100" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33"
I20240501 13:59:14.681128 31140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de38b193e71042be81354e5fab815100" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd"
I20240501 13:59:14.681382 31140 raft_consensus.cc:3037] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.682080 31215 raft_consensus.cc:3037] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.682718 31342 leader_election.cc:288] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 election: Requested vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:14.683023 31141 raft_consensus.cc:2454] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 1.
I20240501 13:59:14.683753 31256 leader_election.cc:302] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters:
I20240501 13:59:14.684159 31376 raft_consensus.cc:2781] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.684127 31214 raft_consensus.cc:2454] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 1.
I20240501 13:59:14.684135 31342 raft_consensus.cc:2781] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.684383 31342 raft_consensus.cc:686] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 1 LEADER]: Becoming Leader. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Running, Role: LEADER
I20240501 13:59:14.684664 31342 consensus_queue.cc:237] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.685642 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73d5a368fc6343c894d83d60e87208b2" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd"
I20240501 13:59:14.685855 31141 raft_consensus.cc:3037] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.687772 31215 raft_consensus.cc:2454] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 1.
I20240501 13:59:14.688390 31141 raft_consensus.cc:2454] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 1.
I20240501 13:59:14.688557 31255 leader_election.cc:302] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7266f74949944f78bb3f0dd836af30ae, de57af3402884bc5afee432f9f25ee33; no voters:
I20240501 13:59:14.688354 31039 catalog_manager.cc:5484] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 7266f74949944f78bb3f0dd836af30ae (127.29.194.195). New cstate: current_term: 1 leader_uuid: "7266f74949944f78bb3f0dd836af30ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:14.689013 31376 raft_consensus.cc:686] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 1 LEADER]: Becoming Leader. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Running, Role: LEADER
I20240501 13:59:14.689391 31376 consensus_queue.cc:237] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.690018 31341 raft_consensus.cc:504] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.691025 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce08af8a10f439c97841daa52a9185c" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd"
I20240501 13:59:14.690999 31341 leader_election.cc:288] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 election: Requested vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.691265 31141 raft_consensus.cc:3037] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.691818 31290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce08af8a10f439c97841daa52a9185c" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae"
I20240501 13:59:14.692087 31140 raft_consensus.cc:2454] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 1.
I20240501 13:59:14.692618 31342 raft_consensus.cc:2781] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.692853 31342 raft_consensus.cc:686] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 1 LEADER]: Becoming Leader. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Running, Role: LEADER
I20240501 13:59:14.693220 31342 consensus_queue.cc:237] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.694600 31141 raft_consensus.cc:2454] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 1.
I20240501 13:59:14.696894 31182 leader_election.cc:302] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, de57af3402884bc5afee432f9f25ee33; no voters: 7266f74949944f78bb3f0dd836af30ae
I20240501 13:59:14.697636 31341 raft_consensus.cc:2781] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.697808 31341 raft_consensus.cc:686] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 1 LEADER]: Becoming Leader. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Running, Role: LEADER
I20240501 13:59:14.698179 31341 consensus_queue.cc:237] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.701390 31374 raft_consensus.cc:504] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.702508 31289 raft_consensus.cc:2379] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de57af3402884bc5afee432f9f25ee33 in current term 1: Already voted for candidate 7266f74949944f78bb3f0dd836af30ae in this term.
I20240501 13:59:14.702870 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce08af8a10f439c97841daa52a9185c" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd"
I20240501 13:59:14.702735 31039 catalog_manager.cc:5484] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 7266f74949944f78bb3f0dd836af30ae (127.29.194.195). New cstate: current_term: 1 leader_uuid: "7266f74949944f78bb3f0dd836af30ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:14.703163 31141 raft_consensus.cc:2379] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7266f74949944f78bb3f0dd836af30ae in current term 1: Already voted for candidate de57af3402884bc5afee432f9f25ee33 in this term.
I20240501 13:59:14.704500 31038 catalog_manager.cc:5484] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 reported cstate change: term changed from 0 to 1, leader changed from <none> to de57af3402884bc5afee432f9f25ee33 (127.29.194.194). New cstate: current_term: 1 leader_uuid: "de57af3402884bc5afee432f9f25ee33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:14.705092 31374 leader_election.cc:288] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 election: Requested vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:14.705562 31215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ce08af8a10f439c97841daa52a9185c" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33"
I20240501 13:59:14.706163 31255 leader_election.cc:302] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7266f74949944f78bb3f0dd836af30ae; no voters: 5f979dcf8ade47c5b1947a66e264addd, de57af3402884bc5afee432f9f25ee33
I20240501 13:59:14.707213 31374 raft_consensus.cc:2726] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 13:59:14.708989 31038 catalog_manager.cc:5484] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 reported cstate change: term changed from 0 to 1, leader changed from <none> to de57af3402884bc5afee432f9f25ee33 (127.29.194.194). New cstate: current_term: 1 leader_uuid: "de57af3402884bc5afee432f9f25ee33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:14.733856 31374 raft_consensus.cc:482] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.734112 31374 raft_consensus.cc:504] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.734827 31374 leader_election.cc:288] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:14.735368 31215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a6fcec53d245a4ac0c66e278405a59" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33" is_pre_election: true
I20240501 13:59:14.735364 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a6fcec53d245a4ac0c66e278405a59" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd" is_pre_election: true
I20240501 13:59:14.735589 31215 raft_consensus.cc:2454] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 0.
I20240501 13:59:14.735658 31141 raft_consensus.cc:2454] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 0.
I20240501 13:59:14.736019 31255 leader_election.cc:302] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7266f74949944f78bb3f0dd836af30ae, de57af3402884bc5afee432f9f25ee33; no voters:
I20240501 13:59:14.736290 31374 raft_consensus.cc:2781] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.736410 31374 raft_consensus.cc:482] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.736513 31374 raft_consensus.cc:3037] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.738577 31374 raft_consensus.cc:504] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.739223 31374 leader_election.cc:288] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 election: Requested vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415)
I20240501 13:59:14.739401 31344 raft_consensus.cc:482] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.739603 31344 raft_consensus.cc:504] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.739792 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a6fcec53d245a4ac0c66e278405a59" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd"
I20240501 13:59:14.739796 31215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0a6fcec53d245a4ac0c66e278405a59" candidate_uuid: "7266f74949944f78bb3f0dd836af30ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33"
I20240501 13:59:14.740022 31141 raft_consensus.cc:3037] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.740077 31215 raft_consensus.cc:3037] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.740335 31344 leader_election.cc:288] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.740911 31289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b80f7c34f4e11a7b352f576e8e744" candidate_uuid: "5f979dcf8ade47c5b1947a66e264addd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae" is_pre_election: true
I20240501 13:59:14.741118 31289 raft_consensus.cc:2454] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f979dcf8ade47c5b1947a66e264addd in term 0.
I20240501 13:59:14.741456 31214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b80f7c34f4e11a7b352f576e8e744" candidate_uuid: "5f979dcf8ade47c5b1947a66e264addd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33" is_pre_election: true
I20240501 13:59:14.741690 31104 leader_election.cc:302] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters:
I20240501 13:59:14.741716 31214 raft_consensus.cc:2454] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f979dcf8ade47c5b1947a66e264addd in term 0.
I20240501 13:59:14.742161 31344 raft_consensus.cc:2781] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.742280 31344 raft_consensus.cc:482] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.742389 31344 raft_consensus.cc:3037] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.742904 31141 raft_consensus.cc:2454] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 1.
I20240501 13:59:14.743274 31256 leader_election.cc:302] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters:
I20240501 13:59:14.743636 31374 raft_consensus.cc:2781] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.743788 31374 raft_consensus.cc:686] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 1 LEADER]: Becoming Leader. State: Replica: 7266f74949944f78bb3f0dd836af30ae, State: Running, Role: LEADER
I20240501 13:59:14.743850 31215 raft_consensus.cc:2454] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7266f74949944f78bb3f0dd836af30ae in term 1.
I20240501 13:59:14.744093 31374 consensus_queue.cc:237] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.745622 31344 raft_consensus.cc:504] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.746443 31344 leader_election.cc:288] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 election: Requested vote from peers de57af3402884bc5afee432f9f25ee33 (127.29.194.194:38415), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.746955 31289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b80f7c34f4e11a7b352f576e8e744" candidate_uuid: "5f979dcf8ade47c5b1947a66e264addd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae"
I20240501 13:59:14.747063 31215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b21b80f7c34f4e11a7b352f576e8e744" candidate_uuid: "5f979dcf8ade47c5b1947a66e264addd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de57af3402884bc5afee432f9f25ee33"
I20240501 13:59:14.747169 31289 raft_consensus.cc:3037] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.747262 31215 raft_consensus.cc:3037] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.747710 31038 catalog_manager.cc:5484] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 7266f74949944f78bb3f0dd836af30ae (127.29.194.195). New cstate: current_term: 1 leader_uuid: "7266f74949944f78bb3f0dd836af30ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:14.750521 31215 raft_consensus.cc:2454] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f979dcf8ade47c5b1947a66e264addd in term 1.
I20240501 13:59:14.750523 31289 raft_consensus.cc:2454] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f979dcf8ade47c5b1947a66e264addd in term 1.
I20240501 13:59:14.750954 31104 leader_election.cc:302] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae; no voters:
I20240501 13:59:14.751245 31344 raft_consensus.cc:2781] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.751533 31344 raft_consensus.cc:686] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 1 LEADER]: Becoming Leader. State: Replica: 5f979dcf8ade47c5b1947a66e264addd, State: Running, Role: LEADER
I20240501 13:59:14.751821 31344 consensus_queue.cc:237] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.754329 31038 catalog_manager.cc:5484] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193). New cstate: current_term: 1 leader_uuid: "5f979dcf8ade47c5b1947a66e264addd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:14.844246 31376 raft_consensus.cc:482] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.844571 31376 raft_consensus.cc:504] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.845266 31376 leader_election.cc:288] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.845952 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c00b157eab43428141fe8508ac9bec" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd" is_pre_election: true
I20240501 13:59:14.845947 31289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5c00b157eab43428141fe8508ac9bec" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae" is_pre_election: true
I20240501 13:59:14.846261 31289 raft_consensus.cc:2379] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de57af3402884bc5afee432f9f25ee33 in current term 1: Already voted for candidate 5f979dcf8ade47c5b1947a66e264addd in this term.
I20240501 13:59:14.846720 31179 leader_election.cc:302] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de57af3402884bc5afee432f9f25ee33; no voters: 5f979dcf8ade47c5b1947a66e264addd, 7266f74949944f78bb3f0dd836af30ae
I20240501 13:59:14.846985 31376 raft_consensus.cc:3037] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.848989 31376 raft_consensus.cc:2726] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 13:59:14.899559 31376 raft_consensus.cc:482] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:14.899787 31376 raft_consensus.cc:504] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.901031 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3db8b744ada4840b5a8237a813211fb" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd" is_pre_election: true
I20240501 13:59:14.901078 31289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3db8b744ada4840b5a8237a813211fb" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae" is_pre_election: true
I20240501 13:59:14.901036 31376 leader_election.cc:288] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.901412 31141 raft_consensus.cc:2454] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 0.
I20240501 13:59:14.901387 31289 raft_consensus.cc:2454] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 0.
I20240501 13:59:14.901823 31182 leader_election.cc:302] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, de57af3402884bc5afee432f9f25ee33; no voters:
I20240501 13:59:14.902057 31376 raft_consensus.cc:2781] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:14.902163 31376 raft_consensus.cc:482] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:14.902251 31376 raft_consensus.cc:3037] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.904103 31376 raft_consensus.cc:504] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.904630 31376 leader_election.cc:288] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 election: Requested vote from peers 5f979dcf8ade47c5b1947a66e264addd (127.29.194.193:42839), 7266f74949944f78bb3f0dd836af30ae (127.29.194.195:41005)
I20240501 13:59:14.905062 31141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3db8b744ada4840b5a8237a813211fb" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f979dcf8ade47c5b1947a66e264addd"
I20240501 13:59:14.905061 31289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3db8b744ada4840b5a8237a813211fb" candidate_uuid: "de57af3402884bc5afee432f9f25ee33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7266f74949944f78bb3f0dd836af30ae"
I20240501 13:59:14.905339 31141 raft_consensus.cc:3037] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.905356 31289 raft_consensus.cc:3037] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:14.908370 31141 raft_consensus.cc:2454] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 1.
I20240501 13:59:14.908392 31289 raft_consensus.cc:2454] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de57af3402884bc5afee432f9f25ee33 in term 1.
I20240501 13:59:14.908880 31182 leader_election.cc:302] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f979dcf8ade47c5b1947a66e264addd, de57af3402884bc5afee432f9f25ee33; no voters:
I20240501 13:59:14.909179 31376 raft_consensus.cc:2781] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:14.909317 31376 raft_consensus.cc:686] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 1 LEADER]: Becoming Leader. State: Replica: de57af3402884bc5afee432f9f25ee33, State: Running, Role: LEADER
I20240501 13:59:14.909579 31376 consensus_queue.cc:237] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } }
I20240501 13:59:14.912108 31038 catalog_manager.cc:5484] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 reported cstate change: term changed from 0 to 1, leader changed from <none> to de57af3402884bc5afee432f9f25ee33 (127.29.194.194). New cstate: current_term: 1 leader_uuid: "de57af3402884bc5afee432f9f25ee33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:14.973210 31344 consensus_queue.cc:1035] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Connected to new peer: Peer: permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:14.978417 31358 consensus_queue.cc:1035] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:14.989251 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:14.993461 31390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.993633 31391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:14.994890 31393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:14.995028 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:14.995510 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:14.995632 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:14.995716 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571954995683 us; error 0 us; skew 500 ppm
I20240501 13:59:14.995950 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:15.003968 30475 webserver.cc:460] Webserver started at http://127.29.194.196:45419/ using document root <none> and password file <none>
I20240501 13:59:15.004252 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:15.004354 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:15.004514 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:15.005304 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-3-root/instance:
uuid: "9d4a39a4514143beb6c3e9330e04b0ef"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-kwk7"
I20240501 13:59:15.007989 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:15.009768 31398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.010196 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 13:59:15.010315 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-3-root
uuid: "9d4a39a4514143beb6c3e9330e04b0ef"
format_stamp: "Formatted at 2024-05-01 13:59:14 on dist-test-slave-kwk7"
I20240501 13:59:15.010454 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714571949891039-30475-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:15.042815 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:15.043493 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:15.044154 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:15.045200 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:15.045297 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.045403 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:15.045486 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:15.064210 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.196:40613
I20240501 13:59:15.064312 31460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.196:40613 every 8 connection(s)
I20240501 13:59:15.071771 31461 heartbeater.cc:347] Connected to a master server at 127.29.194.254:43215
I20240501 13:59:15.071910 31461 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:15.072295 31461 heartbeater.cc:511] Master 127.29.194.254:43215 requested a full tablet report, sending...
I20240501 13:59:15.073173 31038 ts_manager.cc:194] Registered new tserver with Master: 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196:40613)
I20240501 13:59:15.074080 31038 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56302
I20240501 13:59:15.086879 31376 consensus_queue.cc:1035] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.091905 31377 consensus_queue.cc:1035] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.131731 31092 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:15.133950 31092 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:15.134222 31092 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:15.144323 31374 consensus_queue.cc:1035] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.148617 31375 consensus_queue.cc:1035] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.203859 31374 consensus_queue.cc:1035] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.207755 31342 consensus_queue.cc:1035] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.256940 31344 consensus_queue.cc:1035] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Connected to new peer: Peer: permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.261327 31344 consensus_queue.cc:1035] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.263976 31342 consensus_queue.cc:1035] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.269232 31342 consensus_queue.cc:1035] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.304661 31377 consensus_queue.cc:1035] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.307683 31376 consensus_queue.cc:1035] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.323424 31341 consensus_queue.cc:1035] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:15.327714 31341 consensus_queue.cc:1035] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:16.075132 31461 heartbeater.cc:503] Master 127.29.194.254:43215 was elected leader, sending a full tablet report...
I20240501 13:59:17.134652 31092 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:17.136047 31092 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:17.136224 31092 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:19.136775 31092 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:19.138113 31092 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:19.138293 31092 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:20.151294 31289 consensus_queue.cc:237] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } }
I20240501 13:59:20.155254 31141 raft_consensus.cc:1261] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Refusing update from remote peer 7266f74949944f78bb3f0dd836af30ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:20.155587 31215 raft_consensus.cc:1261] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Refusing update from remote peer 7266f74949944f78bb3f0dd836af30ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:20.155982 31484 consensus_queue.cc:1035] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:20.156373 31475 consensus_queue.cc:1035] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:20.159271 31484 raft_consensus.cc:2932] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.159791 31141 raft_consensus.cc:2932] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
W20240501 13:59:20.161885 31256 consensus_peers.cc:485] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae -> Peer 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196:40613): Couldn't send request to peer 9d4a39a4514143beb6c3e9330e04b0ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71d9917dafba40d0a62fc11f2a03ff13. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:20.164527 31215 raft_consensus.cc:2932] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.165863 31038 catalog_manager.cc:5484] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd reported cstate change: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New cstate: current_term: 1 leader_uuid: "7266f74949944f78bb3f0dd836af30ae" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } attrs { replace: true } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.174134 31215 consensus_queue.cc:237] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } }
I20240501 13:59:20.176708 31289 raft_consensus.cc:1261] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Refusing update from remote peer de57af3402884bc5afee432f9f25ee33: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:20.177256 31141 raft_consensus.cc:1261] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Refusing update from remote peer de57af3402884bc5afee432f9f25ee33: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:20.177601 31496 consensus_queue.cc:1035] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:20.177927 31376 consensus_queue.cc:1035] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:20.181295 31496 raft_consensus.cc:2932] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.181957 31289 raft_consensus.cc:2932] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
W20240501 13:59:20.184372 31182 consensus_peers.cc:485] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 -> Peer 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196:40613): Couldn't send request to peer 9d4a39a4514143beb6c3e9330e04b0ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73d5a368fc6343c894d83d60e87208b2. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:20.187767 31038 catalog_manager.cc:5484] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae reported cstate change: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New cstate: current_term: 1 leader_uuid: "de57af3402884bc5afee432f9f25ee33" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.188053 31141 raft_consensus.cc:2932] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.203006 31141 consensus_queue.cc:237] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } attrs { replace: true } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } }
I20240501 13:59:20.205766 31289 raft_consensus.cc:1261] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Refusing update from remote peer 5f979dcf8ade47c5b1947a66e264addd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:20.206353 31504 consensus_queue.cc:1035] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:20.206547 31215 raft_consensus.cc:1261] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Refusing update from remote peer 5f979dcf8ade47c5b1947a66e264addd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:20.207321 31504 consensus_queue.cc:1035] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Connected to new peer: Peer: permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:20.213302 31505 raft_consensus.cc:2932] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } attrs { replace: true } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.213960 31289 raft_consensus.cc:2932] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } attrs { replace: true } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.214582 31215 raft_consensus.cc:2932] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } attrs { replace: true } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
W20240501 13:59:20.218457 31107 consensus_peers.cc:485] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd -> Peer 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196:40613): Couldn't send request to peer 9d4a39a4514143beb6c3e9330e04b0ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b21b80f7c34f4e11a7b352f576e8e744. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:20.218772 31038 catalog_manager.cc:5484] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae reported cstate change: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New cstate: current_term: 1 leader_uuid: "5f979dcf8ade47c5b1947a66e264addd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } attrs { replace: true } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.222095 31141 consensus_queue.cc:237] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } }
I20240501 13:59:20.224725 31215 raft_consensus.cc:1261] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Refusing update from remote peer 5f979dcf8ade47c5b1947a66e264addd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 13:59:20.225090 31107 consensus_peers.cc:485] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd -> Peer 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196:40613): Couldn't send request to peer 9d4a39a4514143beb6c3e9330e04b0ef. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5c00b157eab43428141fe8508ac9bec. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:20.225265 31289 raft_consensus.cc:1261] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Refusing update from remote peer 5f979dcf8ade47c5b1947a66e264addd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:20.225324 31504 consensus_queue.cc:1035] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Connected to new peer: Peer: permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:20.225836 31505 consensus_queue.cc:1035] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:20.228351 31504 raft_consensus.cc:2932] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.228641 31215 raft_consensus.cc:2932] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.229827 31289 raft_consensus.cc:2932] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.233902 31037 catalog_manager.cc:5484] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 reported cstate change: config changed from index -1 to 2, NON_VOTER 9d4a39a4514143beb6c3e9330e04b0ef (127.29.194.196) added. New cstate: current_term: 1 leader_uuid: "5f979dcf8ade47c5b1947a66e264addd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5f979dcf8ade47c5b1947a66e264addd" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 42839 } } peers { permanent_uuid: "de57af3402884bc5afee432f9f25ee33" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38415 } } peers { permanent_uuid: "7266f74949944f78bb3f0dd836af30ae" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41005 } attrs { replace: true } } peers { permanent_uuid: "9d4a39a4514143beb6c3e9330e04b0ef" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 40613 } attrs { promote: true } } }
I20240501 13:59:20.323017 30475 tablet_server.cc:178] TabletServer@127.29.194.193:0 shutting down...
W20240501 13:59:20.326468 31026 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20240501 13:59:20.329927 31087 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 13:59:20.330063 31087 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 13:59:20.332283 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:20.332569 30475 tablet_replica.cc:330] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd: stopping tablet replica
I20240501 13:59:20.332796 30475 raft_consensus.cc:2229] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.333027 30475 raft_consensus.cc:2258] T f3db8b744ada4840b5a8237a813211fb P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.333654 30475 tablet_replica.cc:330] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd: stopping tablet replica
I20240501 13:59:20.333868 30475 raft_consensus.cc:2229] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.334033 30475 raft_consensus.cc:2258] T f0a6fcec53d245a4ac0c66e278405a59 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.334617 30475 tablet_replica.cc:330] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd: stopping tablet replica
I20240501 13:59:20.334774 30475 raft_consensus.cc:2229] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.335244 30475 raft_consensus.cc:2258] T b21b80f7c34f4e11a7b352f576e8e744 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 13:59:20.335271 31087 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 13:59:20.335356 31087 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 13:59:20.335906 30475 tablet_replica.cc:330] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd: stopping tablet replica
I20240501 13:59:20.336131 30475 raft_consensus.cc:2229] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.336457 30475 raft_consensus.cc:2258] T e5c00b157eab43428141fe8508ac9bec P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.337095 30475 tablet_replica.cc:330] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd: stopping tablet replica
I20240501 13:59:20.337275 30475 raft_consensus.cc:2229] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.337534 30475 raft_consensus.cc:2258] T 71d9917dafba40d0a62fc11f2a03ff13 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.338176 30475 tablet_replica.cc:330] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd: stopping tablet replica
I20240501 13:59:20.338335 30475 raft_consensus.cc:2229] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.338482 30475 raft_consensus.cc:2258] T 3ce08af8a10f439c97841daa52a9185c P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.338925 30475 tablet_replica.cc:330] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd: stopping tablet replica
I20240501 13:59:20.339067 30475 raft_consensus.cc:2229] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.339251 30475 raft_consensus.cc:2258] T 73d5a368fc6343c894d83d60e87208b2 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.340059 30475 tablet_replica.cc:330] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd: stopping tablet replica
I20240501 13:59:20.340237 30475 raft_consensus.cc:2229] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.340400 30475 raft_consensus.cc:2258] T de38b193e71042be81354e5fab815100 P 5f979dcf8ade47c5b1947a66e264addd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.357528 30475 tablet_server.cc:194] TabletServer@127.29.194.193:0 shutdown complete.
I20240501 13:59:20.362200 30475 tablet_server.cc:178] TabletServer@127.29.194.194:0 shutting down...
W20240501 13:59:20.365787 31087 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 13:59:20.365908 31087 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 13:59:20.372491 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:20.372808 30475 tablet_replica.cc:330] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33: stopping tablet replica
I20240501 13:59:20.373039 30475 raft_consensus.cc:2229] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.373638 30475 raft_consensus.cc:2258] T f3db8b744ada4840b5a8237a813211fb P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.374352 30475 tablet_replica.cc:330] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33: stopping tablet replica
I20240501 13:59:20.374567 30475 raft_consensus.cc:2229] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.374900 30475 raft_consensus.cc:2258] T 73d5a368fc6343c894d83d60e87208b2 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.375586 30475 tablet_replica.cc:330] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33: stopping tablet replica
I20240501 13:59:20.375772 30475 raft_consensus.cc:2229] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.375945 30475 raft_consensus.cc:2258] T e5c00b157eab43428141fe8508ac9bec P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.376617 30475 tablet_replica.cc:330] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33: stopping tablet replica
I20240501 13:59:20.376813 30475 raft_consensus.cc:2229] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.376996 30475 raft_consensus.cc:2258] T b21b80f7c34f4e11a7b352f576e8e744 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.377645 30475 tablet_replica.cc:330] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33: stopping tablet replica
I20240501 13:59:20.377825 30475 raft_consensus.cc:2229] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.377972 30475 raft_consensus.cc:2258] T f0a6fcec53d245a4ac0c66e278405a59 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.378391 30475 tablet_replica.cc:330] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33: stopping tablet replica
I20240501 13:59:20.378527 30475 raft_consensus.cc:2229] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.378707 30475 raft_consensus.cc:2258] T de38b193e71042be81354e5fab815100 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.379110 30475 tablet_replica.cc:330] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33: stopping tablet replica
I20240501 13:59:20.379257 30475 raft_consensus.cc:2229] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.379424 30475 raft_consensus.cc:2258] T 71d9917dafba40d0a62fc11f2a03ff13 P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.380080 30475 tablet_replica.cc:330] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33: stopping tablet replica
I20240501 13:59:20.380218 30475 raft_consensus.cc:2229] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.380481 30475 raft_consensus.cc:2258] T 3ce08af8a10f439c97841daa52a9185c P de57af3402884bc5afee432f9f25ee33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.386822 30475 tablet_server.cc:194] TabletServer@127.29.194.194:0 shutdown complete.
I20240501 13:59:20.391149 30475 tablet_server.cc:178] TabletServer@127.29.194.195:0 shutting down...
W20240501 13:59:20.394618 31087 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 13:59:20.395255 31087 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 13:59:20.399766 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:20.400007 30475 tablet_replica.cc:330] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae: stopping tablet replica
I20240501 13:59:20.400187 30475 raft_consensus.cc:2229] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.400509 30475 raft_consensus.cc:2258] T f0a6fcec53d245a4ac0c66e278405a59 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.401072 30475 tablet_replica.cc:330] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae: stopping tablet replica
I20240501 13:59:20.401212 30475 raft_consensus.cc:2229] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.401342 30475 raft_consensus.cc:2258] T f3db8b744ada4840b5a8237a813211fb P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.401736 30475 tablet_replica.cc:330] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae: stopping tablet replica
I20240501 13:59:20.401846 30475 raft_consensus.cc:2229] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.402065 30475 raft_consensus.cc:2258] T de38b193e71042be81354e5fab815100 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.402441 30475 tablet_replica.cc:330] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae: stopping tablet replica
I20240501 13:59:20.402563 30475 raft_consensus.cc:2229] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.402817 30475 raft_consensus.cc:2258] T 71d9917dafba40d0a62fc11f2a03ff13 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.403343 30475 tablet_replica.cc:330] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae: stopping tablet replica
I20240501 13:59:20.403455 30475 raft_consensus.cc:2229] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.403574 30475 raft_consensus.cc:2258] T 3ce08af8a10f439c97841daa52a9185c P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.403919 30475 tablet_replica.cc:330] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae: stopping tablet replica
I20240501 13:59:20.404036 30475 raft_consensus.cc:2229] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.404165 30475 raft_consensus.cc:2258] T b21b80f7c34f4e11a7b352f576e8e744 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.404664 30475 tablet_replica.cc:330] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae: stopping tablet replica
I20240501 13:59:20.404807 30475 raft_consensus.cc:2229] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.404966 30475 raft_consensus.cc:2258] T e5c00b157eab43428141fe8508ac9bec P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.405478 30475 tablet_replica.cc:330] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae: stopping tablet replica
I20240501 13:59:20.405593 30475 raft_consensus.cc:2229] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:20.405718 30475 raft_consensus.cc:2258] T 73d5a368fc6343c894d83d60e87208b2 P 7266f74949944f78bb3f0dd836af30ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.410987 30475 tablet_server.cc:194] TabletServer@127.29.194.195:0 shutdown complete.
I20240501 13:59:20.414844 30475 tablet_server.cc:178] TabletServer@127.29.194.196:0 shutting down...
I20240501 13:59:20.423255 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:20.436028 30475 tablet_server.cc:194] TabletServer@127.29.194.196:0 shutdown complete.
I20240501 13:59:20.438832 30475 master.cc:537] Master@127.29.194.254:43215 shutting down...
I20240501 13:59:20.446198 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:20.446449 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:20.446568 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2d07292c75a94b3f9981568363686c9f: stopping tablet replica
I20240501 13:59:20.461380 30475 master.cc:552] Master@127.29.194.254:43215 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7403 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240501 13:59:20.478842 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:46253,127.29.194.253:42029,127.29.194.252:33437
I20240501 13:59:20.479470 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:20.482857 31514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.483357 31513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.483944 31516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.484583 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:20.484901 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:20.484966 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:20.485021 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571960484994 us; error 0 us; skew 500 ppm
I20240501 13:59:20.485164 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:20.486455 30475 webserver.cc:460] Webserver started at http://127.29.194.254:37459/ using document root <none> and password file <none>
I20240501 13:59:20.486662 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:20.486742 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:20.486850 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:20.487483 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "a314d05b63b14be0baf9e55be1675c3e"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.490018 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:20.491647 31521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.492086 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 13:59:20.492208 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "a314d05b63b14be0baf9e55be1675c3e"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.492352 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:20.521056 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:20.521689 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:20.535544 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:46253
I20240501 13:59:20.535601 31572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:46253 every 8 connection(s)
I20240501 13:59:20.536969 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 13:59:20.537011 31573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 13:59:20.540146 31575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.540680 31576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.540645 31573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
W20240501 13:59:20.541215 31578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.541788 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:20.542619 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:20.542693 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:20.542757 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571960542718 us; error 0 us; skew 500 ppm
I20240501 13:59:20.542923 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:20.544380 30475 webserver.cc:460] Webserver started at http://127.29.194.253:35221/ using document root <none> and password file <none>
I20240501 13:59:20.544585 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:20.544715 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:20.544863 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:20.545511 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root/instance:
uuid: "431f9319bbd749ae92d87b681dcbd00e"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.547991 31573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:20.548512 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
W20240501 13:59:20.549897 31573 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.194.253:42029: Network error: Client connection negotiation failed: client connection to 127.29.194.253:42029: connect: Connection refused (error 111)
I20240501 13:59:20.550356 31586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.550670 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 13:59:20.550778 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root
uuid: "431f9319bbd749ae92d87b681dcbd00e"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.550925 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:20.580241 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:20.580153 31573 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } attempt: 1
I20240501 13:59:20.581043 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240501 13:59:20.582149 31573 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.194.253:42029: Network error: Client connection negotiation failed: client connection to 127.29.194.253:42029: connect: Connection refused (error 111)
I20240501 13:59:20.592599 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.253:42029
I20240501 13:59:20.592644 31637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.253:42029 every 8 connection(s)
I20240501 13:59:20.594106 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 13:59:20.594187 31638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 13:59:20.597291 31640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.597906 31641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.598369 31643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.598809 31638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:20.598999 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:20.599292 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:20.599427 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:20.599539 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571960599478 us; error 0 us; skew 500 ppm
I20240501 13:59:20.599789 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:20.601325 30475 webserver.cc:460] Webserver started at http://127.29.194.252:45719/ using document root <none> and password file <none>
I20240501 13:59:20.601539 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:20.601635 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:20.601765 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:20.602432 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root/instance:
uuid: "22015ae910924aca9885d45568460b4d"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.605192 31638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:20.605244 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20240501 13:59:20.607471 31650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.607915 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 13:59:20.608021 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root
uuid: "22015ae910924aca9885d45568460b4d"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.608172 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:20.612303 31638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
W20240501 13:59:20.614239 31638 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.29.194.252:33437: Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111)
I20240501 13:59:20.626389 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:20.627009 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:20.640388 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.252:33437
I20240501 13:59:20.640436 31702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.252:33437 every 8 connection(s)
I20240501 13:59:20.641474 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:59:20.641512 31573 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } attempt: 2
I20240501 13:59:20.642064 31703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:20.644520 31703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:20.647936 31573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:20.651887 31703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:20.657955 31703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } has no permanent_uuid. Determining permanent_uuid...
I20240501 13:59:20.658269 31573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e: Bootstrap starting.
I20240501 13:59:20.660030 31573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:20.662004 31573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e: No bootstrap required, opened a new log
I20240501 13:59:20.662809 31573 raft_consensus.cc:348] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:20.663130 31573 raft_consensus.cc:374] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:20.663225 31573 raft_consensus.cc:729] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a314d05b63b14be0baf9e55be1675c3e, State: Initialized, Role: FOLLOWER
I20240501 13:59:20.663499 31573 consensus_queue.cc:260] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:20.664125 31708 sys_catalog.cc:455] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:20.664305 31573 sys_catalog.cc:564] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:20.664356 31708 sys_catalog.cc:458] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:20.666160 31703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d: Bootstrap starting.
I20240501 13:59:20.667979 31703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:20.669517 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240501 13:59:20.670122 31703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d: No bootstrap required, opened a new log
W20240501 13:59:20.670845 31720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 13:59:20.670984 31720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 13:59:20.670975 31703 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:20.671207 31703 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:20.671264 31703 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22015ae910924aca9885d45568460b4d, State: Initialized, Role: FOLLOWER
I20240501 13:59:20.671473 31703 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:20.672041 31721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:20.672259 31721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:20.672360 31703 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:20.672456 31638 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } attempt: 1
W20240501 13:59:20.679090 31732 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 13:59:20.679215 31732 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 13:59:20.681274 31638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: Bootstrap starting.
I20240501 13:59:20.682425 31638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:20.683940 31638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: No bootstrap required, opened a new log
I20240501 13:59:20.685060 31638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:20.685237 31638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:20.685312 31638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431f9319bbd749ae92d87b681dcbd00e, State: Initialized, Role: FOLLOWER
I20240501 13:59:20.685552 31638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:20.686111 31734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:20.686308 31734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:20.686306 31638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:20.689627 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 13:59:20.690521 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:20.690905 31745 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 13:59:20.691018 31745 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 13:59:20.694370 31746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.694887 31747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.695362 31749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.695928 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:20.696154 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:20.696224 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:20.696265 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571960696252 us; error 0 us; skew 500 ppm
I20240501 13:59:20.696399 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:20.697573 30475 webserver.cc:460] Webserver started at http://127.29.194.193:42727/ using document root <none> and password file <none>
I20240501 13:59:20.697798 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:20.697870 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:20.697994 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:20.698637 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "2c38deee4cf949c4aa6ca9ff53f2a628"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.701007 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 13:59:20.702605 31754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.702944 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 13:59:20.703053 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "2c38deee4cf949c4aa6ca9ff53f2a628"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.703166 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:20.721416 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:20.722005 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:20.722613 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:20.723541 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:20.723621 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.723718 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:20.723798 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.740737 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:44781
I20240501 13:59:20.740772 31816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:44781 every 8 connection(s)
I20240501 13:59:20.744123 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:20.753855 31824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.754119 31825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.754443 31818 heartbeater.cc:347] Connected to a master server at 127.29.194.254:46253
I20240501 13:59:20.754534 31817 heartbeater.cc:347] Connected to a master server at 127.29.194.252:33437
I20240501 13:59:20.754642 31818 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.755255 31818 heartbeater.cc:511] Master 127.29.194.254:46253 requested a full tablet report, sending...
I20240501 13:59:20.754662 31817 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.756330 31817 heartbeater.cc:511] Master 127.29.194.252:33437 requested a full tablet report, sending...
I20240501 13:59:20.756774 31538 ts_manager.cc:194] Registered new tserver with Master: 2c38deee4cf949c4aa6ca9ff53f2a628 (127.29.194.193:44781)
I20240501 13:59:20.758178 31668 ts_manager.cc:194] Registered new tserver with Master: 2c38deee4cf949c4aa6ca9ff53f2a628 (127.29.194.193:44781)
W20240501 13:59:20.762365 31827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.762913 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:20.763177 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:20.763276 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:20.763362 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571960763321 us; error 0 us; skew 500 ppm
I20240501 13:59:20.763535 31819 heartbeater.cc:347] Connected to a master server at 127.29.194.253:42029
I20240501 13:59:20.763540 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:20.763681 31819 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.764129 31819 heartbeater.cc:511] Master 127.29.194.253:42029 requested a full tablet report, sending...
I20240501 13:59:20.765188 30475 webserver.cc:460] Webserver started at http://127.29.194.194:36553/ using document root <none> and password file <none>
I20240501 13:59:20.765494 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:20.765539 31603 ts_manager.cc:194] Registered new tserver with Master: 2c38deee4cf949c4aa6ca9ff53f2a628 (127.29.194.193:44781)
I20240501 13:59:20.765627 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:20.765790 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:20.766469 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "0537496cb7344d68955e08ded7837eef"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.768899 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:20.770937 31832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.771343 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 13:59:20.771438 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "0537496cb7344d68955e08ded7837eef"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.771560 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:20.782881 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:20.783428 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:20.783970 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:20.784914 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:20.784986 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.785095 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:20.785148 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.802842 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:35593
I20240501 13:59:20.802878 31894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:35593 every 8 connection(s)
I20240501 13:59:20.805325 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:20.815948 31902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:20.817152 31903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.819576 31895 heartbeater.cc:347] Connected to a master server at 127.29.194.252:33437
I20240501 13:59:20.819715 31895 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.820271 31895 heartbeater.cc:511] Master 127.29.194.252:33437 requested a full tablet report, sending...
I20240501 13:59:20.820677 31896 heartbeater.cc:347] Connected to a master server at 127.29.194.254:46253
I20240501 13:59:20.820849 31896 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.821305 31896 heartbeater.cc:511] Master 127.29.194.254:46253 requested a full tablet report, sending...
I20240501 13:59:20.822746 31668 ts_manager.cc:194] Registered new tserver with Master: 0537496cb7344d68955e08ded7837eef (127.29.194.194:35593)
I20240501 13:59:20.823689 31538 ts_manager.cc:194] Registered new tserver with Master: 0537496cb7344d68955e08ded7837eef (127.29.194.194:35593)
I20240501 13:59:20.824391 31897 heartbeater.cc:347] Connected to a master server at 127.29.194.253:42029
I20240501 13:59:20.824502 31897 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.824941 31897 heartbeater.cc:511] Master 127.29.194.253:42029 requested a full tablet report, sending...
W20240501 13:59:20.825248 31905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:20.825814 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:20.825892 31603 ts_manager.cc:194] Registered new tserver with Master: 0537496cb7344d68955e08ded7837eef (127.29.194.194:35593)
I20240501 13:59:20.826159 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:20.826244 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:20.826308 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571960826282 us; error 0 us; skew 500 ppm
I20240501 13:59:20.826454 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:20.827632 30475 webserver.cc:460] Webserver started at http://127.29.194.195:37351/ using document root <none> and password file <none>
I20240501 13:59:20.827867 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:20.827936 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:20.828043 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:20.828606 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "a72337f280cb4dd3aa8195298786e78d"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.830987 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:20.832643 31910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.833004 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 13:59:20.833109 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "a72337f280cb4dd3aa8195298786e78d"
format_stamp: "Formatted at 2024-05-01 13:59:20 on dist-test-slave-kwk7"
I20240501 13:59:20.833202 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:20.844208 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:20.844811 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:20.845379 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:20.846187 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:20.846266 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.846333 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:20.846390 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:20.865156 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:41929
I20240501 13:59:20.865195 31972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:41929 every 8 connection(s)
I20240501 13:59:20.875452 31973 heartbeater.cc:347] Connected to a master server at 127.29.194.252:33437
I20240501 13:59:20.875653 31973 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.876055 31975 heartbeater.cc:347] Connected to a master server at 127.29.194.253:42029
I20240501 13:59:20.876091 31974 heartbeater.cc:347] Connected to a master server at 127.29.194.254:46253
I20240501 13:59:20.876236 31975 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.876293 31973 heartbeater.cc:511] Master 127.29.194.252:33437 requested a full tablet report, sending...
I20240501 13:59:20.876317 31974 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:20.876803 31975 heartbeater.cc:511] Master 127.29.194.253:42029 requested a full tablet report, sending...
I20240501 13:59:20.877014 31974 heartbeater.cc:511] Master 127.29.194.254:46253 requested a full tablet report, sending...
I20240501 13:59:20.877382 31668 ts_manager.cc:194] Registered new tserver with Master: a72337f280cb4dd3aa8195298786e78d (127.29.194.195:41929)
I20240501 13:59:20.877879 31603 ts_manager.cc:194] Registered new tserver with Master: a72337f280cb4dd3aa8195298786e78d (127.29.194.195:41929)
I20240501 13:59:20.878086 31538 ts_manager.cc:194] Registered new tserver with Master: a72337f280cb4dd3aa8195298786e78d (127.29.194.195:41929)
I20240501 13:59:20.878599 30475 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011371146s
I20240501 13:59:21.000360 31721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:21.000566 31721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:21.001273 31721 leader_election.cc:288] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a314d05b63b14be0baf9e55be1675c3e (127.29.194.254:46253), 431f9319bbd749ae92d87b681dcbd00e (127.29.194.253:42029)
I20240501 13:59:21.001793 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22015ae910924aca9885d45568460b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a314d05b63b14be0baf9e55be1675c3e" is_pre_election: true
I20240501 13:59:21.001888 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22015ae910924aca9885d45568460b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431f9319bbd749ae92d87b681dcbd00e" is_pre_election: true
I20240501 13:59:21.002018 31548 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22015ae910924aca9885d45568460b4d in term 0.
I20240501 13:59:21.002133 31613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22015ae910924aca9885d45568460b4d in term 0.
I20240501 13:59:21.002473 31651 leader_election.cc:302] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22015ae910924aca9885d45568460b4d, a314d05b63b14be0baf9e55be1675c3e; no voters:
I20240501 13:59:21.002750 31721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:21.002859 31721 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:21.002950 31721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.005049 31721 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:21.005589 31721 leader_election.cc:288] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [CANDIDATE]: Term 1 election: Requested vote from peers a314d05b63b14be0baf9e55be1675c3e (127.29.194.254:46253), 431f9319bbd749ae92d87b681dcbd00e (127.29.194.253:42029)
I20240501 13:59:21.006580 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22015ae910924aca9885d45568460b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431f9319bbd749ae92d87b681dcbd00e"
I20240501 13:59:21.006623 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22015ae910924aca9885d45568460b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a314d05b63b14be0baf9e55be1675c3e"
I20240501 13:59:21.006760 31613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.006808 31548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:21.009004 31613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22015ae910924aca9885d45568460b4d in term 1.
I20240501 13:59:21.009034 31548 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22015ae910924aca9885d45568460b4d in term 1.
I20240501 13:59:21.009413 31651 leader_election.cc:302] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22015ae910924aca9885d45568460b4d, 431f9319bbd749ae92d87b681dcbd00e; no voters:
I20240501 13:59:21.009716 31721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:21.010079 31721 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 1 LEADER]: Becoming Leader. State: Replica: 22015ae910924aca9885d45568460b4d, State: Running, Role: LEADER
I20240501 13:59:21.010334 31721 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:21.011677 31981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22015ae910924aca9885d45568460b4d. Latest consensus state: current_term: 1 leader_uuid: "22015ae910924aca9885d45568460b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:21.012006 31981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:21.012615 31983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:21.013957 31983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:21.016743 31613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 1 FOLLOWER]: Refusing update from remote peer 22015ae910924aca9885d45568460b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:21.016768 31548 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 1 FOLLOWER]: Refusing update from remote peer 22015ae910924aca9885d45568460b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:21.017339 31721 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:21.017549 31981 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:21.021282 31734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22015ae910924aca9885d45568460b4d. Latest consensus state: current_term: 1 leader_uuid: "22015ae910924aca9885d45568460b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:21.021512 31734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:21.022599 31708 sys_catalog.cc:455] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22015ae910924aca9885d45568460b4d. Latest consensus state: current_term: 1 leader_uuid: "22015ae910924aca9885d45568460b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:21.022887 31708 sys_catalog.cc:458] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:21.023515 31981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22015ae910924aca9885d45568460b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:21.023742 31981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:21.024451 31981 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "22015ae910924aca9885d45568460b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:21.024988 31981 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:21.025332 31983 catalog_manager.cc:1347] Generated new cluster ID: 0e3ef6d873114bd29f09ac4440ec5f64
I20240501 13:59:21.025496 31983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:21.025990 31734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22015ae910924aca9885d45568460b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:21.026268 31734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:21.027187 31708 sys_catalog.cc:455] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "22015ae910924aca9885d45568460b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:21.027403 31708 sys_catalog.cc:458] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:21.049013 31983 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:21.050040 31983 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:21.060042 31983 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d: Generated new TSK 0
I20240501 13:59:21.060354 31983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:21.672384 31720 catalog_manager.cc:1259] Loaded cluster ID: 0e3ef6d873114bd29f09ac4440ec5f64
I20240501 13:59:21.672489 31720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e: loading cluster ID for follower catalog manager: success
I20240501 13:59:21.674425 31720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e: acquiring CA information for follower catalog manager: success
I20240501 13:59:21.675441 31720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 13:59:21.692523 31745 catalog_manager.cc:1259] Loaded cluster ID: 0e3ef6d873114bd29f09ac4440ec5f64
I20240501 13:59:21.692631 31745 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: loading cluster ID for follower catalog manager: success
I20240501 13:59:21.694706 31745 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: acquiring CA information for follower catalog manager: success
I20240501 13:59:21.695755 31745 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 13:59:21.761664 31668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58116
I20240501 13:59:21.825570 31668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58126
I20240501 13:59:21.880127 31668 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58140
I20240501 13:59:22.678257 31731 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:22.762826 31817 heartbeater.cc:503] Master 127.29.194.252:33437 was elected leader, sending a full tablet report...
I20240501 13:59:22.826822 31895 heartbeater.cc:503] Master 127.29.194.252:33437 was elected leader, sending a full tablet report...
I20240501 13:59:22.881248 31973 heartbeater.cc:503] Master 127.29.194.252:33437 was elected leader, sending a full tablet report...
I20240501 13:59:23.140167 30475 test_util.cc:268] Using random seed: -456259538
I20240501 13:59:23.156850 31668 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58152:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 13:59:23.199479 31782 tablet_service.cc:1467] Processing CreateTablet for tablet ed7fe25f184848618fc9447985df3595 (DEFAULT_TABLE table=test-workload [id=3fda060b562643d48cab57851eef5232]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 13:59:23.200049 31782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed7fe25f184848618fc9447985df3595. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.201400 31860 tablet_service.cc:1467] Processing CreateTablet for tablet 1b51da752f7e46e4a9725c3e956c5194 (DEFAULT_TABLE table=test-workload [id=3fda060b562643d48cab57851eef5232]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 13:59:23.201922 31860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b51da752f7e46e4a9725c3e956c5194. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.202493 31938 tablet_service.cc:1467] Processing CreateTablet for tablet 29beb302447f49e783e832fd9f27e3b0 (DEFAULT_TABLE table=test-workload [id=3fda060b562643d48cab57851eef5232]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 13:59:23.202906 31938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29beb302447f49e783e832fd9f27e3b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:23.208410 32020 tablet_bootstrap.cc:492] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d: Bootstrap starting.
I20240501 13:59:23.208804 32021 tablet_bootstrap.cc:492] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef: Bootstrap starting.
I20240501 13:59:23.210692 32021 tablet_bootstrap.cc:654] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.211244 32020 tablet_bootstrap.cc:654] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.212659 32022 tablet_bootstrap.cc:492] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628: Bootstrap starting.
I20240501 13:59:23.213528 32021 tablet_bootstrap.cc:492] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef: No bootstrap required, opened a new log
I20240501 13:59:23.213711 32021 ts_tablet_manager.cc:1367] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 13:59:23.213924 32020 tablet_bootstrap.cc:492] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d: No bootstrap required, opened a new log
I20240501 13:59:23.214087 32020 ts_tablet_manager.cc:1367] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 13:59:23.215009 32021 raft_consensus.cc:348] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0537496cb7344d68955e08ded7837eef" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 35593 } }
I20240501 13:59:23.215178 32022 tablet_bootstrap.cc:654] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:23.215093 32020 raft_consensus.cc:348] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72337f280cb4dd3aa8195298786e78d" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41929 } }
I20240501 13:59:23.215234 32021 raft_consensus.cc:374] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.215276 32020 raft_consensus.cc:374] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.215440 32020 raft_consensus.cc:729] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72337f280cb4dd3aa8195298786e78d, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.215415 32021 raft_consensus.cc:729] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0537496cb7344d68955e08ded7837eef, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.215684 32020 consensus_queue.cc:260] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72337f280cb4dd3aa8195298786e78d" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41929 } }
I20240501 13:59:23.215873 32020 raft_consensus.cc:388] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:23.215775 32021 consensus_queue.cc:260] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0537496cb7344d68955e08ded7837eef" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 35593 } }
I20240501 13:59:23.215983 32020 raft_consensus.cc:482] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:23.216066 32021 raft_consensus.cc:388] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:23.216114 32020 raft_consensus.cc:3037] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.216220 32021 raft_consensus.cc:482] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:23.216434 32021 raft_consensus.cc:3037] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.217396 32022 tablet_bootstrap.cc:492] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628: No bootstrap required, opened a new log
I20240501 13:59:23.217602 32022 ts_tablet_manager.cc:1367] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240501 13:59:23.218427 32022 raft_consensus.cc:348] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c38deee4cf949c4aa6ca9ff53f2a628" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 44781 } }
I20240501 13:59:23.218500 32020 raft_consensus.cc:504] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72337f280cb4dd3aa8195298786e78d" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41929 } }
I20240501 13:59:23.218693 32022 raft_consensus.cc:374] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:23.218840 32022 raft_consensus.cc:729] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c38deee4cf949c4aa6ca9ff53f2a628, State: Initialized, Role: FOLLOWER
I20240501 13:59:23.218815 32020 leader_election.cc:302] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a72337f280cb4dd3aa8195298786e78d; no voters:
I20240501 13:59:23.219174 32022 consensus_queue.cc:260] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c38deee4cf949c4aa6ca9ff53f2a628" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 44781 } }
I20240501 13:59:23.219512 32022 raft_consensus.cc:388] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:23.219534 32020 leader_election.cc:288] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:23.219697 32022 raft_consensus.cc:482] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:23.219727 32026 raft_consensus.cc:2781] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:23.219908 32022 raft_consensus.cc:3037] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:23.220105 32021 raft_consensus.cc:504] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0537496cb7344d68955e08ded7837eef" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 35593 } }
I20240501 13:59:23.220602 32021 leader_election.cc:302] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0537496cb7344d68955e08ded7837eef; no voters:
I20240501 13:59:23.223747 32026 raft_consensus.cc:686] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 1 LEADER]: Becoming Leader. State: Replica: a72337f280cb4dd3aa8195298786e78d, State: Running, Role: LEADER
I20240501 13:59:23.224241 32026 consensus_queue.cc:237] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72337f280cb4dd3aa8195298786e78d" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41929 } }
I20240501 13:59:23.224375 32022 raft_consensus.cc:504] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c38deee4cf949c4aa6ca9ff53f2a628" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 44781 } }
I20240501 13:59:23.224845 32022 leader_election.cc:302] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c38deee4cf949c4aa6ca9ff53f2a628; no voters:
I20240501 13:59:23.227396 32029 raft_consensus.cc:2781] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:23.227622 32021 leader_election.cc:288] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:23.228122 32020 ts_tablet_manager.cc:1398] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d: Time spent starting tablet: real 0.014s user 0.003s sys 0.010s
I20240501 13:59:23.230481 31668 catalog_manager.cc:5484] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d reported cstate change: term changed from 0 to 1, leader changed from <none> to a72337f280cb4dd3aa8195298786e78d (127.29.194.195). New cstate: current_term: 1 leader_uuid: "a72337f280cb4dd3aa8195298786e78d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72337f280cb4dd3aa8195298786e78d" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 41929 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:23.232970 32022 leader_election.cc:288] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:23.233444 32031 raft_consensus.cc:2781] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:23.234042 32022 ts_tablet_manager.cc:1398] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20240501 13:59:23.234562 32031 raft_consensus.cc:686] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 1 LEADER]: Becoming Leader. State: Replica: 2c38deee4cf949c4aa6ca9ff53f2a628, State: Running, Role: LEADER
I20240501 13:59:23.234938 32031 consensus_queue.cc:237] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c38deee4cf949c4aa6ca9ff53f2a628" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 44781 } }
I20240501 13:59:23.244969 32029 raft_consensus.cc:686] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 1 LEADER]: Becoming Leader. State: Replica: 0537496cb7344d68955e08ded7837eef, State: Running, Role: LEADER
I20240501 13:59:23.245530 31668 catalog_manager.cc:5484] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c38deee4cf949c4aa6ca9ff53f2a628 (127.29.194.193). New cstate: current_term: 1 leader_uuid: "2c38deee4cf949c4aa6ca9ff53f2a628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c38deee4cf949c4aa6ca9ff53f2a628" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 44781 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:23.258096 32029 consensus_queue.cc:237] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0537496cb7344d68955e08ded7837eef" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 35593 } }
I20240501 13:59:23.262457 32021 ts_tablet_manager.cc:1398] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef: Time spent starting tablet: real 0.049s user 0.007s sys 0.012s
I20240501 13:59:23.266010 31667 catalog_manager.cc:5484] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef reported cstate change: term changed from 0 to 1, leader changed from <none> to 0537496cb7344d68955e08ded7837eef (127.29.194.194). New cstate: current_term: 1 leader_uuid: "0537496cb7344d68955e08ded7837eef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0537496cb7344d68955e08ded7837eef" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 35593 } health_report { overall_health: HEALTHY } } }
I20240501 13:59:23.279466 30475 master.cc:537] Master@127.29.194.252:33437 shutting down...
I20240501 13:59:23.285992 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:23.286355 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:23.286465 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22015ae910924aca9885d45568460b4d: stopping tablet replica
I20240501 13:59:23.301471 30475 master.cc:552] Master@127.29.194.252:33437 shutdown complete.
W20240501 13:59:24.251205 31973 heartbeater.cc:644] Failed to heartbeat to 127.29.194.252:33437 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111)
I20240501 13:59:24.777655 32045 raft_consensus.cc:482] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22015ae910924aca9885d45568460b4d)
I20240501 13:59:24.777655 32046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 22015ae910924aca9885d45568460b4d)
I20240501 13:59:24.777937 32046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:24.777912 32045 raft_consensus.cc:504] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:24.778718 32045 leader_election.cc:288] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 431f9319bbd749ae92d87b681dcbd00e (127.29.194.253:42029), 22015ae910924aca9885d45568460b4d (127.29.194.252:33437)
I20240501 13:59:24.778766 32046 leader_election.cc:288] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a314d05b63b14be0baf9e55be1675c3e (127.29.194.254:46253), 22015ae910924aca9885d45568460b4d (127.29.194.252:33437)
I20240501 13:59:24.779951 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "431f9319bbd749ae92d87b681dcbd00e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a314d05b63b14be0baf9e55be1675c3e" is_pre_election: true
I20240501 13:59:24.780206 31548 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431f9319bbd749ae92d87b681dcbd00e in term 1.
I20240501 13:59:24.780157 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a314d05b63b14be0baf9e55be1675c3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "431f9319bbd749ae92d87b681dcbd00e" is_pre_election: true
I20240501 13:59:24.780871 31587 leader_election.cc:302] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431f9319bbd749ae92d87b681dcbd00e, a314d05b63b14be0baf9e55be1675c3e; no voters:
I20240501 13:59:24.780969 31613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a314d05b63b14be0baf9e55be1675c3e in term 1.
W20240501 13:59:24.781236 31522 leader_election.cc:334] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 22015ae910924aca9885d45568460b4d (127.29.194.252:33437): Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111)
I20240501 13:59:24.781515 31522 leader_election.cc:302] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 431f9319bbd749ae92d87b681dcbd00e, a314d05b63b14be0baf9e55be1675c3e; no voters: 22015ae910924aca9885d45568460b4d
I20240501 13:59:24.781587 32046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240501 13:59:24.781756 31587 leader_election.cc:334] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 22015ae910924aca9885d45568460b4d (127.29.194.252:33437): Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111)
I20240501 13:59:24.781809 32045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 13:59:24.781919 32045 raft_consensus.cc:482] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22015ae910924aca9885d45568460b4d)
I20240501 13:59:24.782023 32045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:24.782034 32046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 22015ae910924aca9885d45568460b4d)
I20240501 13:59:24.782264 32046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:24.788904 32045 raft_consensus.cc:504] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:24.788964 32046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:24.789587 32045 leader_election.cc:288] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [CANDIDATE]: Term 2 election: Requested vote from peers 431f9319bbd749ae92d87b681dcbd00e (127.29.194.253:42029), 22015ae910924aca9885d45568460b4d (127.29.194.252:33437)
I20240501 13:59:24.789733 32046 leader_election.cc:288] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 2 election: Requested vote from peers a314d05b63b14be0baf9e55be1675c3e (127.29.194.254:46253), 22015ae910924aca9885d45568460b4d (127.29.194.252:33437)
I20240501 13:59:24.790501 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "431f9319bbd749ae92d87b681dcbd00e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a314d05b63b14be0baf9e55be1675c3e"
I20240501 13:59:24.790529 31613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a314d05b63b14be0baf9e55be1675c3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "431f9319bbd749ae92d87b681dcbd00e"
I20240501 13:59:24.790819 31613 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a314d05b63b14be0baf9e55be1675c3e in current term 2: Already voted for candidate 431f9319bbd749ae92d87b681dcbd00e in this term.
I20240501 13:59:24.790822 31548 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 431f9319bbd749ae92d87b681dcbd00e in current term 2: Already voted for candidate a314d05b63b14be0baf9e55be1675c3e in this term.
W20240501 13:59:24.791347 31587 leader_election.cc:334] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 22015ae910924aca9885d45568460b4d (127.29.194.252:33437): Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111)
I20240501 13:59:24.791636 31587 leader_election.cc:302] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 431f9319bbd749ae92d87b681dcbd00e; no voters: 22015ae910924aca9885d45568460b4d, a314d05b63b14be0baf9e55be1675c3e
I20240501 13:59:24.791965 32046 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20240501 13:59:24.791987 31522 leader_election.cc:334] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 22015ae910924aca9885d45568460b4d (127.29.194.252:33437): Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111)
I20240501 13:59:24.792119 31522 leader_election.cc:302] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a314d05b63b14be0baf9e55be1675c3e; no voters: 22015ae910924aca9885d45568460b4d, 431f9319bbd749ae92d87b681dcbd00e
I20240501 13:59:24.792397 32045 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 13:59:26.711267 32057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:26.711463 32057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:26.712126 32057 leader_election.cc:288] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a314d05b63b14be0baf9e55be1675c3e (127.29.194.254:46253), 22015ae910924aca9885d45568460b4d (127.29.194.252:33437)
I20240501 13:59:26.712740 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "431f9319bbd749ae92d87b681dcbd00e" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a314d05b63b14be0baf9e55be1675c3e" is_pre_election: true
I20240501 13:59:26.713001 31548 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 431f9319bbd749ae92d87b681dcbd00e in term 2.
W20240501 13:59:26.713223 31587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 13:59:26.717039 31587 leader_election.cc:302] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 431f9319bbd749ae92d87b681dcbd00e, a314d05b63b14be0baf9e55be1675c3e; no voters:
I20240501 13:59:26.717319 32057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 13:59:26.717455 32057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:26.717563 32057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 2 FOLLOWER]: Advancing to term 3
W20240501 13:59:26.717860 31587 leader_election.cc:334] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 22015ae910924aca9885d45568460b4d (127.29.194.252:33437): Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111)
I20240501 13:59:26.719511 32057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:26.720031 32057 leader_election.cc:288] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 3 election: Requested vote from peers a314d05b63b14be0baf9e55be1675c3e (127.29.194.254:46253), 22015ae910924aca9885d45568460b4d (127.29.194.252:33437)
I20240501 13:59:26.720443 31548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "431f9319bbd749ae92d87b681dcbd00e" candidate_term: 3 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "a314d05b63b14be0baf9e55be1675c3e"
I20240501 13:59:26.720647 31548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 2 FOLLOWER]: Advancing to term 3
W20240501 13:59:26.721244 31587 leader_election.cc:334] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 22015ae910924aca9885d45568460b4d (127.29.194.252:33437): Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111)
I20240501 13:59:26.722618 31548 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 431f9319bbd749ae92d87b681dcbd00e in term 3.
I20240501 13:59:26.722960 31587 leader_election.cc:302] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 431f9319bbd749ae92d87b681dcbd00e, a314d05b63b14be0baf9e55be1675c3e; no voters: 22015ae910924aca9885d45568460b4d
I20240501 13:59:26.723214 32057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 13:59:26.723621 32057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 3 LEADER]: Becoming Leader. State: Replica: 431f9319bbd749ae92d87b681dcbd00e, State: Running, Role: LEADER
I20240501 13:59:26.723904 32057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } }
I20240501 13:59:26.724985 32060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431f9319bbd749ae92d87b681dcbd00e. Latest consensus state: current_term: 3 leader_uuid: "431f9319bbd749ae92d87b681dcbd00e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:26.725272 32060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:26.725782 32062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:26.727540 32062 catalog_manager.cc:669] Loaded metadata for table test-workload [id=3fda060b562643d48cab57851eef5232]
I20240501 13:59:26.729005 32062 tablet_loader.cc:96] loaded metadata for tablet 1b51da752f7e46e4a9725c3e956c5194 (table test-workload [id=3fda060b562643d48cab57851eef5232])
I20240501 13:59:26.729367 32062 tablet_loader.cc:96] loaded metadata for tablet 29beb302447f49e783e832fd9f27e3b0 (table test-workload [id=3fda060b562643d48cab57851eef5232])
I20240501 13:59:26.729681 32062 tablet_loader.cc:96] loaded metadata for tablet ed7fe25f184848618fc9447985df3595 (table test-workload [id=3fda060b562643d48cab57851eef5232])
I20240501 13:59:26.729918 32062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:26.730767 32062 catalog_manager.cc:1259] Loaded cluster ID: 0e3ef6d873114bd29f09ac4440ec5f64
I20240501 13:59:26.730871 32062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:26.731973 32062 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:26.732950 32062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: Loaded TSK: 0
I20240501 13:59:26.733618 32062 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 13:59:27.133395 31587 consensus_peers.cc:485] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e -> Peer 22015ae910924aca9885d45568460b4d (127.29.194.252:33437): Couldn't send request to peer 22015ae910924aca9885d45568460b4d. Status: Network error: Client connection negotiation failed: client connection to 127.29.194.252:33437: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:27.236162 31548 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 3 FOLLOWER]: Refusing update from remote peer 431f9319bbd749ae92d87b681dcbd00e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240501 13:59:27.236822 32060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 13:59:27.238623 32063 sys_catalog.cc:455] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431f9319bbd749ae92d87b681dcbd00e. Latest consensus state: current_term: 3 leader_uuid: "431f9319bbd749ae92d87b681dcbd00e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:27.238833 32063 sys_catalog.cc:458] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:27.240870 32066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "431f9319bbd749ae92d87b681dcbd00e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:27.241272 32066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:27.241571 32063 sys_catalog.cc:455] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "431f9319bbd749ae92d87b681dcbd00e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a314d05b63b14be0baf9e55be1675c3e" member_type: VOTER last_known_addr { host: "127.29.194.254" port: 46253 } } peers { permanent_uuid: "431f9319bbd749ae92d87b681dcbd00e" member_type: VOTER last_known_addr { host: "127.29.194.253" port: 42029 } } peers { permanent_uuid: "22015ae910924aca9885d45568460b4d" member_type: VOTER last_known_addr { host: "127.29.194.252" port: 33437 } } }
I20240501 13:59:27.241848 32063 sys_catalog.cc:458] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [sys.catalog]: This master's current role is: FOLLOWER
I20240501 13:59:28.242872 31975 heartbeater.cc:503] Master 127.29.194.253:42029 was elected leader, sending a full tablet report...
I20240501 13:59:28.257709 31819 heartbeater.cc:503] Master 127.29.194.253:42029 was elected leader, sending a full tablet report...
I20240501 13:59:28.276371 31897 heartbeater.cc:503] Master 127.29.194.253:42029 was elected leader, sending a full tablet report...
I20240501 13:59:28.690699 31744 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:28.690865 31744 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:28.854947 30475 tablet_server.cc:178] TabletServer@127.29.194.193:0 shutting down...
I20240501 13:59:28.863476 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:28.863780 30475 tablet_replica.cc:330] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628: stopping tablet replica
I20240501 13:59:28.863986 30475 raft_consensus.cc:2229] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:28.864182 30475 raft_consensus.cc:2258] T ed7fe25f184848618fc9447985df3595 P 2c38deee4cf949c4aa6ca9ff53f2a628 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:28.877717 30475 tablet_server.cc:194] TabletServer@127.29.194.193:0 shutdown complete.
I20240501 13:59:28.880530 30475 tablet_server.cc:178] TabletServer@127.29.194.194:0 shutting down...
I20240501 13:59:28.888968 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:28.889233 30475 tablet_replica.cc:330] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef: stopping tablet replica
I20240501 13:59:28.889426 30475 raft_consensus.cc:2229] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:28.889588 30475 raft_consensus.cc:2258] T 1b51da752f7e46e4a9725c3e956c5194 P 0537496cb7344d68955e08ded7837eef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:28.902806 30475 tablet_server.cc:194] TabletServer@127.29.194.194:0 shutdown complete.
I20240501 13:59:28.905916 30475 tablet_server.cc:178] TabletServer@127.29.194.195:0 shutting down...
I20240501 13:59:28.914865 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:28.915169 30475 tablet_replica.cc:330] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d: stopping tablet replica
I20240501 13:59:28.915370 30475 raft_consensus.cc:2229] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:28.915577 30475 raft_consensus.cc:2258] T 29beb302447f49e783e832fd9f27e3b0 P a72337f280cb4dd3aa8195298786e78d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:28.929121 30475 tablet_server.cc:194] TabletServer@127.29.194.195:0 shutdown complete.
I20240501 13:59:28.931963 30475 master.cc:537] Master@127.29.194.254:46253 shutting down...
I20240501 13:59:28.939637 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:28.939888 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:28.940019 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P a314d05b63b14be0baf9e55be1675c3e: stopping tablet replica
W20240501 13:59:28.946215 31587 consensus_peers.cc:485] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e -> Peer a314d05b63b14be0baf9e55be1675c3e (127.29.194.254:46253): Couldn't send request to peer a314d05b63b14be0baf9e55be1675c3e. Status: Network error: Client connection negotiation failed: client connection to 127.29.194.254:46253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:28.953478 30475 master.cc:552] Master@127.29.194.254:46253 shutdown complete.
I20240501 13:59:28.957208 30475 master.cc:537] Master@127.29.194.253:42029 shutting down...
I20240501 13:59:28.964387 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 3 LEADER]: Raft consensus shutting down.
I20240501 13:59:28.964752 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:28.964864 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 431f9319bbd749ae92d87b681dcbd00e: stopping tablet replica
I20240501 13:59:28.978137 30475 master.cc:552] Master@127.29.194.253:42029 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (8511 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240501 13:59:28.990577 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:43207
I20240501 13:59:28.991119 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:28.994431 32078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.994997 32077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:28.995630 32080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:28.996102 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:28.996376 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:28.996443 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:28.996520 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571968996486 us; error 0 us; skew 500 ppm
I20240501 13:59:28.996680 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:29.003901 30475 webserver.cc:460] Webserver started at http://127.29.194.254:43443/ using document root <none> and password file <none>
I20240501 13:59:29.004145 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:29.004246 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:29.004395 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:29.005376 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "2edc832e9c7d4909bbf078254a0fa970"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-kwk7"
I20240501 13:59:29.007951 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 13:59:29.009629 32085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.010088 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 13:59:29.010210 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "2edc832e9c7d4909bbf078254a0fa970"
format_stamp: "Formatted at 2024-05-01 13:59:28 on dist-test-slave-kwk7"
I20240501 13:59:29.010361 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:29.034951 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:29.035688 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:29.048990 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:43207
I20240501 13:59:29.049026 32136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:43207 every 8 connection(s)
I20240501 13:59:29.050277 32137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:29.054385 32137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970: Bootstrap starting.
I20240501 13:59:29.055651 32137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:29.057209 32137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970: No bootstrap required, opened a new log
I20240501 13:59:29.057816 32137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edc832e9c7d4909bbf078254a0fa970" member_type: VOTER }
I20240501 13:59:29.057948 32137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:29.058046 32137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2edc832e9c7d4909bbf078254a0fa970, State: Initialized, Role: FOLLOWER
I20240501 13:59:29.058274 32137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edc832e9c7d4909bbf078254a0fa970" member_type: VOTER }
I20240501 13:59:29.058423 32137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:29.058507 32137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:29.058614 32137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:29.060284 32137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edc832e9c7d4909bbf078254a0fa970" member_type: VOTER }
I20240501 13:59:29.060503 32137 leader_election.cc:302] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2edc832e9c7d4909bbf078254a0fa970; no voters:
I20240501 13:59:29.060915 32137 leader_election.cc:288] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:29.061067 32140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:29.061579 32140 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 1 LEADER]: Becoming Leader. State: Replica: 2edc832e9c7d4909bbf078254a0fa970, State: Running, Role: LEADER
I20240501 13:59:29.061728 32137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:29.061872 32140 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edc832e9c7d4909bbf078254a0fa970" member_type: VOTER }
I20240501 13:59:29.062901 32142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2edc832e9c7d4909bbf078254a0fa970. Latest consensus state: current_term: 1 leader_uuid: "2edc832e9c7d4909bbf078254a0fa970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edc832e9c7d4909bbf078254a0fa970" member_type: VOTER } }
I20240501 13:59:29.063153 32142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:29.064283 32141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2edc832e9c7d4909bbf078254a0fa970" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2edc832e9c7d4909bbf078254a0fa970" member_type: VOTER } }
I20240501 13:59:29.064457 32141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:29.064918 32147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:29.066501 32147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:29.066932 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:59:29.069669 32147 catalog_manager.cc:1347] Generated new cluster ID: 3486cba34e914cbbb1da62c8fb4bb94f
I20240501 13:59:29.069785 32147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:29.078998 32147 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:29.079747 32147 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:29.090222 32147 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970: Generated new TSK 0
I20240501 13:59:29.090469 32147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:29.099792 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:29.103073 32158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.103756 32159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.104290 32161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:29.104903 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:29.105176 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:29.105259 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:29.105319 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571969105304 us; error 0 us; skew 500 ppm
I20240501 13:59:29.105504 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:29.106748 30475 webserver.cc:460] Webserver started at http://127.29.194.193:43503/ using document root <none> and password file <none>
I20240501 13:59:29.106992 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:29.107071 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:29.107223 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:29.107870 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "4482b523ee4441bd82289235f259aa65"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-kwk7"
I20240501 13:59:29.110456 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 13:59:29.112116 32166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.112496 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 13:59:29.112612 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "4482b523ee4441bd82289235f259aa65"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-kwk7"
I20240501 13:59:29.112766 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:29.118474 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:29.118986 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:29.119493 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:29.120359 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:29.120442 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.120539 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:29.120618 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.134892 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:46665
I20240501 13:59:29.134917 32228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:46665 every 8 connection(s)
I20240501 13:59:29.136765 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:29.140376 32233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.140890 32234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:29.143389 32229 heartbeater.cc:347] Connected to a master server at 127.29.194.254:43207
I20240501 13:59:29.143512 32229 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:29.143882 32229 heartbeater.cc:511] Master 127.29.194.254:43207 requested a full tablet report, sending...
I20240501 13:59:29.144912 32102 ts_manager.cc:194] Registered new tserver with Master: 4482b523ee4441bd82289235f259aa65 (127.29.194.193:46665)
I20240501 13:59:29.146405 32102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50864
W20240501 13:59:29.147024 32236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:29.147526 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:29.147769 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:29.147832 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:29.147909 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571969147877 us; error 0 us; skew 500 ppm
I20240501 13:59:29.148125 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:29.149456 30475 webserver.cc:460] Webserver started at http://127.29.194.194:41609/ using document root <none> and password file <none>
I20240501 13:59:29.149667 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:29.149761 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:29.149890 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:29.150503 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "f191e94b435c4186b6048e9c87fbdfeb"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-kwk7"
I20240501 13:59:29.152788 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20240501 13:59:29.154345 32241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.154731 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 13:59:29.154827 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "f191e94b435c4186b6048e9c87fbdfeb"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-kwk7"
I20240501 13:59:29.154944 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:29.171299 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:29.171846 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:29.172864 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:29.173725 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:29.173789 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.173871 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:29.173914 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.187978 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:38133
I20240501 13:59:29.187999 32303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:38133 every 8 connection(s)
I20240501 13:59:29.189551 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:29.193199 32307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.193706 32308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:29.194290 32310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:29.194859 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:29.195137 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:29.195236 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:29.195290 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571969195269 us; error 0 us; skew 500 ppm
I20240501 13:59:29.195493 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:29.196702 32304 heartbeater.cc:347] Connected to a master server at 127.29.194.254:43207
I20240501 13:59:29.196843 32304 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:29.197049 30475 webserver.cc:460] Webserver started at http://127.29.194.195:33891/ using document root <none> and password file <none>
I20240501 13:59:29.197240 32304 heartbeater.cc:511] Master 127.29.194.254:43207 requested a full tablet report, sending...
I20240501 13:59:29.197326 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:29.197424 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:29.197588 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:29.198637 32102 ts_manager.cc:194] Registered new tserver with Master: f191e94b435c4186b6048e9c87fbdfeb (127.29.194.194:38133)
I20240501 13:59:29.198871 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "2a8d3013d7cc4ee7a904f9e5a4334708"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-kwk7"
I20240501 13:59:29.199671 32102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50874
I20240501 13:59:29.201709 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:29.203306 32315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.203680 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 13:59:29.203801 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "2a8d3013d7cc4ee7a904f9e5a4334708"
format_stamp: "Formatted at 2024-05-01 13:59:29 on dist-test-slave-kwk7"
I20240501 13:59:29.203913 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:29.248653 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:29.249145 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:29.249604 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:29.250399 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:29.250453 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.250524 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:29.250558 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:29.264590 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:43179
I20240501 13:59:29.264632 32377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:43179 every 8 connection(s)
I20240501 13:59:29.271239 32378 heartbeater.cc:347] Connected to a master server at 127.29.194.254:43207
I20240501 13:59:29.271359 32378 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:29.271657 32378 heartbeater.cc:511] Master 127.29.194.254:43207 requested a full tablet report, sending...
I20240501 13:59:29.272429 32102 ts_manager.cc:194] Registered new tserver with Master: 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195:43179)
I20240501 13:59:29.273137 32102 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50878
I20240501 13:59:29.273451 30475 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007897083s
I20240501 13:59:30.147660 32229 heartbeater.cc:503] Master 127.29.194.254:43207 was elected leader, sending a full tablet report...
I20240501 13:59:30.200899 32304 heartbeater.cc:503] Master 127.29.194.254:43207 was elected leader, sending a full tablet report...
I20240501 13:59:30.274246 32378 heartbeater.cc:503] Master 127.29.194.254:43207 was elected leader, sending a full tablet report...
I20240501 13:59:30.300829 30475 test_util.cc:268] Using random seed: -449098858
I20240501 13:59:30.314239 32102 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50886:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 13:59:30.314921 32102 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 13:59:30.347816 32268 tablet_service.cc:1467] Processing CreateTablet for tablet 8c02ea2c5d074862a9286ec08fcb2cb4 (DEFAULT_TABLE table=test-workload [id=01b8881933dd411eb270f6e3c79c76c0]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 13:59:30.347816 32269 tablet_service.cc:1467] Processing CreateTablet for tablet 43772016e716482385d60c4b01a919e5 (DEFAULT_TABLE table=test-workload [id=01b8881933dd411eb270f6e3c79c76c0]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 13:59:30.348337 32268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c02ea2c5d074862a9286ec08fcb2cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:30.348590 32269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43772016e716482385d60c4b01a919e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:30.353102 32194 tablet_service.cc:1467] Processing CreateTablet for tablet 43772016e716482385d60c4b01a919e5 (DEFAULT_TABLE table=test-workload [id=01b8881933dd411eb270f6e3c79c76c0]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 13:59:30.353308 32343 tablet_service.cc:1467] Processing CreateTablet for tablet 43772016e716482385d60c4b01a919e5 (DEFAULT_TABLE table=test-workload [id=01b8881933dd411eb270f6e3c79c76c0]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 13:59:30.353425 32342 tablet_service.cc:1467] Processing CreateTablet for tablet 8c02ea2c5d074862a9286ec08fcb2cb4 (DEFAULT_TABLE table=test-workload [id=01b8881933dd411eb270f6e3c79c76c0]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 13:59:30.353552 32194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43772016e716482385d60c4b01a919e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:30.353643 32343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43772016e716482385d60c4b01a919e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:30.353840 32342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c02ea2c5d074862a9286ec08fcb2cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:30.359779 32193 tablet_service.cc:1467] Processing CreateTablet for tablet 8c02ea2c5d074862a9286ec08fcb2cb4 (DEFAULT_TABLE table=test-workload [id=01b8881933dd411eb270f6e3c79c76c0]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 13:59:30.360164 32193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c02ea2c5d074862a9286ec08fcb2cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:30.361095 32398 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb: Bootstrap starting.
I20240501 13:59:30.362637 32398 tablet_bootstrap.cc:654] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:30.365342 32398 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb: No bootstrap required, opened a new log
I20240501 13:59:30.365593 32398 ts_tablet_manager.cc:1367] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 13:59:30.366537 32398 raft_consensus.cc:348] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.366955 32398 raft_consensus.cc:374] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:30.367082 32398 raft_consensus.cc:729] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f191e94b435c4186b6048e9c87fbdfeb, State: Initialized, Role: FOLLOWER
I20240501 13:59:30.367413 32398 consensus_queue.cc:260] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.368391 32400 tablet_bootstrap.cc:492] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65: Bootstrap starting.
I20240501 13:59:30.368733 32398 ts_tablet_manager.cc:1398] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 13:59:30.369000 32398 tablet_bootstrap.cc:492] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb: Bootstrap starting.
I20240501 13:59:30.369069 32399 tablet_bootstrap.cc:492] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708: Bootstrap starting.
I20240501 13:59:30.370178 32400 tablet_bootstrap.cc:654] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:30.371011 32399 tablet_bootstrap.cc:654] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:30.371867 32398 tablet_bootstrap.cc:654] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:30.374603 32400 tablet_bootstrap.cc:492] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65: No bootstrap required, opened a new log
I20240501 13:59:30.374738 32400 ts_tablet_manager.cc:1367] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20240501 13:59:30.374845 32398 tablet_bootstrap.cc:492] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb: No bootstrap required, opened a new log
I20240501 13:59:30.374958 32398 ts_tablet_manager.cc:1367] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 13:59:30.375365 32400 raft_consensus.cc:348] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.375420 32398 raft_consensus.cc:348] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.375566 32400 raft_consensus.cc:374] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:30.375577 32398 raft_consensus.cc:374] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:30.375641 32400 raft_consensus.cc:729] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4482b523ee4441bd82289235f259aa65, State: Initialized, Role: FOLLOWER
I20240501 13:59:30.375643 32398 raft_consensus.cc:729] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f191e94b435c4186b6048e9c87fbdfeb, State: Initialized, Role: FOLLOWER
I20240501 13:59:30.375900 32398 consensus_queue.cc:260] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.375900 32400 consensus_queue.cc:260] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.376459 32398 ts_tablet_manager.cc:1398] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 13:59:30.376627 32400 ts_tablet_manager.cc:1398] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:30.377038 32400 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65: Bootstrap starting.
I20240501 13:59:30.378795 32400 tablet_bootstrap.cc:654] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:30.380334 32399 tablet_bootstrap.cc:492] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708: No bootstrap required, opened a new log
I20240501 13:59:30.380497 32399 ts_tablet_manager.cc:1367] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 13:59:30.381147 32399 raft_consensus.cc:348] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.381284 32399 raft_consensus.cc:374] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:30.381317 32399 raft_consensus.cc:729] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8d3013d7cc4ee7a904f9e5a4334708, State: Initialized, Role: FOLLOWER
I20240501 13:59:30.381479 32399 consensus_queue.cc:260] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.383236 32399 ts_tablet_manager.cc:1398] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 13:59:30.383491 32399 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: Bootstrap starting.
I20240501 13:59:30.384892 32399 tablet_bootstrap.cc:654] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:30.386543 32399 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: No bootstrap required, opened a new log
I20240501 13:59:30.386644 32399 ts_tablet_manager.cc:1367] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 13:59:30.387217 32399 raft_consensus.cc:348] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.387354 32399 raft_consensus.cc:374] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:30.387409 32399 raft_consensus.cc:729] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a8d3013d7cc4ee7a904f9e5a4334708, State: Initialized, Role: FOLLOWER
I20240501 13:59:30.387598 32399 consensus_queue.cc:260] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.387986 32399 ts_tablet_manager.cc:1398] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 13:59:30.389645 32400 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65: No bootstrap required, opened a new log
I20240501 13:59:30.389783 32400 ts_tablet_manager.cc:1367] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.007s
I20240501 13:59:30.390420 32400 raft_consensus.cc:348] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.390640 32400 raft_consensus.cc:374] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:30.390738 32400 raft_consensus.cc:729] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4482b523ee4441bd82289235f259aa65, State: Initialized, Role: FOLLOWER
I20240501 13:59:30.390940 32400 consensus_queue.cc:260] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.391454 32400 ts_tablet_manager.cc:1398] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 13:59:30.397296 32402 raft_consensus.cc:482] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:30.397423 32402 raft_consensus.cc:504] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.398128 32402 leader_election.cc:288] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195:43179), 4482b523ee4441bd82289235f259aa65 (127.29.194.193:46665)
I20240501 13:59:30.404006 32353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43772016e716482385d60c4b01a919e5" candidate_uuid: "f191e94b435c4186b6048e9c87fbdfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" is_pre_election: true
I20240501 13:59:30.404008 32204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43772016e716482385d60c4b01a919e5" candidate_uuid: "f191e94b435c4186b6048e9c87fbdfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4482b523ee4441bd82289235f259aa65" is_pre_election: true
I20240501 13:59:30.404273 32353 raft_consensus.cc:2454] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f191e94b435c4186b6048e9c87fbdfeb in term 0.
I20240501 13:59:30.404304 32204 raft_consensus.cc:2454] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f191e94b435c4186b6048e9c87fbdfeb in term 0.
I20240501 13:59:30.404749 32242 leader_election.cc:302] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a8d3013d7cc4ee7a904f9e5a4334708, 4482b523ee4441bd82289235f259aa65, f191e94b435c4186b6048e9c87fbdfeb; no voters:
I20240501 13:59:30.405002 32402 raft_consensus.cc:2781] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:30.405081 32402 raft_consensus.cc:482] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:30.405161 32402 raft_consensus.cc:3037] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:30.408454 32402 raft_consensus.cc:504] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.409065 32402 leader_election.cc:288] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [CANDIDATE]: Term 1 election: Requested vote from peers 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195:43179), 4482b523ee4441bd82289235f259aa65 (127.29.194.193:46665)
I20240501 13:59:30.409392 32353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43772016e716482385d60c4b01a919e5" candidate_uuid: "f191e94b435c4186b6048e9c87fbdfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708"
I20240501 13:59:30.409466 32204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43772016e716482385d60c4b01a919e5" candidate_uuid: "f191e94b435c4186b6048e9c87fbdfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4482b523ee4441bd82289235f259aa65"
I20240501 13:59:30.409535 32353 raft_consensus.cc:3037] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:30.409636 32204 raft_consensus.cc:3037] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:30.410552 32402 raft_consensus.cc:482] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 13:59:30.410692 32402 raft_consensus.cc:504] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.411814 32203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c02ea2c5d074862a9286ec08fcb2cb4" candidate_uuid: "f191e94b435c4186b6048e9c87fbdfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4482b523ee4441bd82289235f259aa65" is_pre_election: true
I20240501 13:59:30.412035 32203 raft_consensus.cc:2454] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f191e94b435c4186b6048e9c87fbdfeb in term 0.
I20240501 13:59:30.412240 32402 leader_election.cc:288] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4482b523ee4441bd82289235f259aa65 (127.29.194.193:46665), 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195:43179)
I20240501 13:59:30.412358 32244 leader_election.cc:302] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4482b523ee4441bd82289235f259aa65, f191e94b435c4186b6048e9c87fbdfeb; no voters:
I20240501 13:59:30.412425 32352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c02ea2c5d074862a9286ec08fcb2cb4" candidate_uuid: "f191e94b435c4186b6048e9c87fbdfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" is_pre_election: true
I20240501 13:59:30.412582 32402 raft_consensus.cc:2781] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 13:59:30.412611 32352 raft_consensus.cc:2454] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f191e94b435c4186b6048e9c87fbdfeb in term 0.
I20240501 13:59:30.412662 32402 raft_consensus.cc:482] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 13:59:30.412772 32402 raft_consensus.cc:3037] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:30.412920 32204 raft_consensus.cc:2454] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f191e94b435c4186b6048e9c87fbdfeb in term 1.
I20240501 13:59:30.413228 32244 leader_election.cc:302] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4482b523ee4441bd82289235f259aa65, f191e94b435c4186b6048e9c87fbdfeb; no voters:
I20240501 13:59:30.414520 32353 raft_consensus.cc:2454] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f191e94b435c4186b6048e9c87fbdfeb in term 1.
I20240501 13:59:30.415086 32412 raft_consensus.cc:2781] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:30.415499 32412 raft_consensus.cc:686] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Becoming Leader. State: Replica: f191e94b435c4186b6048e9c87fbdfeb, State: Running, Role: LEADER
I20240501 13:59:30.415750 32412 consensus_queue.cc:237] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:30.418252 32101 catalog_manager.cc:5484] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb reported cstate change: term changed from 0 to 1, leader changed from <none> to f191e94b435c4186b6048e9c87fbdfeb (127.29.194.194). New cstate: current_term: 1 leader_uuid: "f191e94b435c4186b6048e9c87fbdfeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:30.421530 32402 raft_consensus.cc:504] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.422011 32402 leader_election.cc:288] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [CANDIDATE]: Term 1 election: Requested vote from peers 4482b523ee4441bd82289235f259aa65 (127.29.194.193:46665), 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195:43179)
I20240501 13:59:30.422343 32204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c02ea2c5d074862a9286ec08fcb2cb4" candidate_uuid: "f191e94b435c4186b6048e9c87fbdfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4482b523ee4441bd82289235f259aa65"
I20240501 13:59:30.422494 32204 raft_consensus.cc:3037] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:30.422453 32353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c02ea2c5d074862a9286ec08fcb2cb4" candidate_uuid: "f191e94b435c4186b6048e9c87fbdfeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708"
I20240501 13:59:30.422608 32353 raft_consensus.cc:3037] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:30.426668 32353 raft_consensus.cc:2454] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f191e94b435c4186b6048e9c87fbdfeb in term 1.
I20240501 13:59:30.426664 32204 raft_consensus.cc:2454] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f191e94b435c4186b6048e9c87fbdfeb in term 1.
I20240501 13:59:30.427081 32244 leader_election.cc:302] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a8d3013d7cc4ee7a904f9e5a4334708, 4482b523ee4441bd82289235f259aa65, f191e94b435c4186b6048e9c87fbdfeb; no voters:
I20240501 13:59:30.427361 32402 raft_consensus.cc:2781] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:30.427485 32402 raft_consensus.cc:686] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Becoming Leader. State: Replica: f191e94b435c4186b6048e9c87fbdfeb, State: Running, Role: LEADER
I20240501 13:59:30.427685 32402 consensus_queue.cc:237] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } }
I20240501 13:59:30.429951 32101 catalog_manager.cc:5484] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb reported cstate change: term changed from 0 to 1, leader changed from <none> to f191e94b435c4186b6048e9c87fbdfeb (127.29.194.194). New cstate: current_term: 1 leader_uuid: "f191e94b435c4186b6048e9c87fbdfeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:30.892889 32402 consensus_queue.cc:1035] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:30.897240 32412 consensus_queue.cc:1035] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:30.965953 32413 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:30.969609 32402 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 13:59:31.066581 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:31.073257 32279 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "43772016e716482385d60c4b01a919e5"
dest_uuid: "f191e94b435c4186b6048e9c87fbdfeb"
mode: GRACEFUL
new_leader_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708"
from {username='slave'} at 127.0.0.1:60944
I20240501 13:59:31.073539 32279 raft_consensus.cc:595] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Received request to transfer leadership to TS 2a8d3013d7cc4ee7a904f9e5a4334708
I20240501 13:59:31.074065 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 13:59:31.074210 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:31.436797 32413 raft_consensus.cc:979] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb: : Instructing follower 2a8d3013d7cc4ee7a904f9e5a4334708 to start an election
I20240501 13:59:31.436998 32402 raft_consensus.cc:1067] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Signalling peer 2a8d3013d7cc4ee7a904f9e5a4334708 to start an election
I20240501 13:59:31.438020 32353 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "43772016e716482385d60c4b01a919e5"
dest_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708"
from {username='slave'} at 127.0.0.1:34662
I20240501 13:59:31.438228 32353 raft_consensus.cc:482] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 13:59:31.438316 32353 raft_consensus.cc:3037] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:31.508826 32353 raft_consensus.cc:504] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:31.510818 32353 leader_election.cc:288] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [CANDIDATE]: Term 2 election: Requested vote from peers f191e94b435c4186b6048e9c87fbdfeb (127.29.194.194:38133), 4482b523ee4441bd82289235f259aa65 (127.29.194.193:46665)
I20240501 13:59:31.516088 32279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43772016e716482385d60c4b01a919e5" candidate_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f191e94b435c4186b6048e9c87fbdfeb"
I20240501 13:59:31.516292 32279 raft_consensus.cc:3032] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 13:59:31.516363 32279 raft_consensus.cc:729] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f191e94b435c4186b6048e9c87fbdfeb, State: Running, Role: LEADER
I20240501 13:59:31.516724 32279 consensus_queue.cc:260] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:31.517094 32279 raft_consensus.cc:3037] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:31.517282 32204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43772016e716482385d60c4b01a919e5" candidate_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4482b523ee4441bd82289235f259aa65"
I20240501 13:59:31.517426 32204 raft_consensus.cc:3037] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Advancing to term 2
I20240501 13:59:31.613631 32279 raft_consensus.cc:2454] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8d3013d7cc4ee7a904f9e5a4334708 in term 2.
I20240501 13:59:31.613631 32204 raft_consensus.cc:2454] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a8d3013d7cc4ee7a904f9e5a4334708 in term 2.
I20240501 13:59:31.614276 32319 leader_election.cc:302] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a8d3013d7cc4ee7a904f9e5a4334708, f191e94b435c4186b6048e9c87fbdfeb; no voters:
I20240501 13:59:31.614890 32430 raft_consensus.cc:2781] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 13:59:31.615260 32430 raft_consensus.cc:686] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 2 LEADER]: Becoming Leader. State: Replica: 2a8d3013d7cc4ee7a904f9e5a4334708, State: Running, Role: LEADER
I20240501 13:59:31.615579 32430 consensus_queue.cc:237] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } }
I20240501 13:59:31.617789 32102 catalog_manager.cc:5484] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 reported cstate change: term changed from 1 to 2, leader changed from f191e94b435c4186b6048e9c87fbdfeb (127.29.194.194) to 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195). New cstate: current_term: 2 leader_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:32.052541 32279 raft_consensus.cc:1261] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 2 FOLLOWER]: Refusing update from remote peer 2a8d3013d7cc4ee7a904f9e5a4334708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:59:32.053136 32430 consensus_queue.cc:1035] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:32.056825 32204 raft_consensus.cc:1261] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 2 FOLLOWER]: Refusing update from remote peer 2a8d3013d7cc4ee7a904f9e5a4334708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 13:59:32.057349 32430 consensus_queue.cc:1035] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:33.074522 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:33.075039 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:33.075153 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:35.075554 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:35.076148 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:35.076303 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:37.076671 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:37.077337 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:37.077474 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:39.077893 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:39.078485 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:39.078619 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:41.078948 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:41.079432 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:41.079545 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:41.494987 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:41.498993 32444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:41.499605 32445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:41.500074 32447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:41.500777 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:41.501055 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:41.501137 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:41.501209 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571981501176 us; error 0 us; skew 500 ppm
I20240501 13:59:41.501377 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:41.502696 30475 webserver.cc:460] Webserver started at http://127.29.194.196:34529/ using document root <none> and password file <none>
I20240501 13:59:41.502918 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:41.503021 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:41.503175 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:41.503909 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-3-root/instance:
uuid: "82f81984814f44eea5eafef3ebdf7156"
format_stamp: "Formatted at 2024-05-01 13:59:41 on dist-test-slave-kwk7"
I20240501 13:59:41.506484 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20240501 13:59:41.508044 32452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:41.508374 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 13:59:41.508471 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-3-root
uuid: "82f81984814f44eea5eafef3ebdf7156"
format_stamp: "Formatted at 2024-05-01 13:59:41 on dist-test-slave-kwk7"
I20240501 13:59:41.508579 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:41.527153 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:41.527693 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:41.528283 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 13:59:41.529155 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 13:59:41.529222 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:41.529296 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 13:59:41.529340 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:41.545537 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.196:36515
I20240501 13:59:41.545558 32514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.196:36515 every 8 connection(s)
I20240501 13:59:41.552502 32515 heartbeater.cc:347] Connected to a master server at 127.29.194.254:43207
I20240501 13:59:41.552637 32515 heartbeater.cc:464] Registering TS with master...
I20240501 13:59:41.552992 32515 heartbeater.cc:511] Master 127.29.194.254:43207 requested a full tablet report, sending...
I20240501 13:59:41.553781 32101 ts_manager.cc:194] Registered new tserver with Master: 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196:36515)
I20240501 13:59:41.554666 32101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48694
I20240501 13:59:42.087343 32279 consensus_queue.cc:237] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: true } }
I20240501 13:59:42.089970 32203 raft_consensus.cc:1261] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Refusing update from remote peer f191e94b435c4186b6048e9c87fbdfeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:42.089999 32353 raft_consensus.cc:1261] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Refusing update from remote peer f191e94b435c4186b6048e9c87fbdfeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 13:59:42.090831 32522 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:42.091235 32443 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 13:59:42.094141 32443 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: true } } }
I20240501 13:59:42.095046 32353 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: true } } }
I20240501 13:59:42.095167 32203 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: true } } }
W20240501 13:59:42.098682 32243 consensus_peers.cc:485] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb -> Peer 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196:36515): Couldn't send request to peer 82f81984814f44eea5eafef3ebdf7156. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c02ea2c5d074862a9286ec08fcb2cb4. This is attempt 1: this message will repeat every 5th retry.
I20240501 13:59:42.100059 32102 catalog_manager.cc:5484] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb reported cstate change: config changed from index -1 to 2, NON_VOTER 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) added. New cstate: current_term: 1 leader_uuid: "f191e94b435c4186b6048e9c87fbdfeb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:42.555773 32515 heartbeater.cc:503] Master 127.29.194.254:43207 was elected leader, sending a full tablet report...
I20240501 13:59:42.581763 32532 ts_tablet_manager.cc:902] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: Initiating tablet copy from peer f191e94b435c4186b6048e9c87fbdfeb (127.29.194.194:38133)
I20240501 13:59:42.583782 32532 tablet_copy_client.cc:308] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.194:38133
I20240501 13:59:42.592130 32289 tablet_copy_service.cc:140] P f191e94b435c4186b6048e9c87fbdfeb: Received BeginTabletCopySession request for tablet 8c02ea2c5d074862a9286ec08fcb2cb4 from peer 82f81984814f44eea5eafef3ebdf7156 ({username='slave'} at 127.0.0.1:41894)
I20240501 13:59:42.592295 32289 tablet_copy_service.cc:161] P f191e94b435c4186b6048e9c87fbdfeb: Beginning new tablet copy session on tablet 8c02ea2c5d074862a9286ec08fcb2cb4 from peer 82f81984814f44eea5eafef3ebdf7156 at {username='slave'} at 127.0.0.1:41894: session id = 82f81984814f44eea5eafef3ebdf7156-8c02ea2c5d074862a9286ec08fcb2cb4
I20240501 13:59:42.595238 32289 tablet_copy_source_session.cc:205] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 13:59:42.597136 32532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c02ea2c5d074862a9286ec08fcb2cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:42.602332 32532 tablet_copy_client.cc:783] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: tablet copy: Starting download of 0 data blocks...
I20240501 13:59:42.602720 32532 tablet_copy_client.cc:647] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: tablet copy: Starting download of 1 WAL segments...
I20240501 13:59:42.604555 32532 tablet_copy_client.cc:523] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 13:59:42.608004 32532 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: Bootstrap starting.
I20240501 13:59:42.615057 32532 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 13:59:42.615514 32532 tablet_bootstrap.cc:492] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: Bootstrap complete.
I20240501 13:59:42.615736 32532 ts_tablet_manager.cc:1367] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20240501 13:59:42.616355 32532 raft_consensus.cc:348] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: true } }
I20240501 13:59:42.616604 32532 raft_consensus.cc:729] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 82f81984814f44eea5eafef3ebdf7156, State: Initialized, Role: LEARNER
I20240501 13:59:42.616855 32532 consensus_queue.cc:260] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: true } }
I20240501 13:59:42.617623 32532 ts_tablet_manager.cc:1398] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240501 13:59:42.619294 32289 tablet_copy_service.cc:342] P f191e94b435c4186b6048e9c87fbdfeb: Request end of tablet copy session 82f81984814f44eea5eafef3ebdf7156-8c02ea2c5d074862a9286ec08fcb2cb4 received from {username='slave'} at 127.0.0.1:41894
I20240501 13:59:42.619412 32289 tablet_copy_service.cc:433] P f191e94b435c4186b6048e9c87fbdfeb: ending tablet copy session 82f81984814f44eea5eafef3ebdf7156-8c02ea2c5d074862a9286ec08fcb2cb4 on tablet 8c02ea2c5d074862a9286ec08fcb2cb4 with peer 82f81984814f44eea5eafef3ebdf7156
I20240501 13:59:43.047093 32490 raft_consensus.cc:1203] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 13:59:43.079839 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:43.080515 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:43.080646 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:43.607972 32542 raft_consensus.cc:1050] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb: attempting to promote NON_VOTER 82f81984814f44eea5eafef3ebdf7156 to VOTER
I20240501 13:59:43.608975 32542 consensus_queue.cc:237] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } }
I20240501 13:59:43.611349 32490 raft_consensus.cc:1261] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 LEARNER]: Refusing update from remote peer f191e94b435c4186b6048e9c87fbdfeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:59:43.611382 32353 raft_consensus.cc:1261] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Refusing update from remote peer f191e94b435c4186b6048e9c87fbdfeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:59:43.611893 32544 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:59:43.612169 32203 raft_consensus.cc:1261] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Refusing update from remote peer f191e94b435c4186b6048e9c87fbdfeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 13:59:43.612746 32540 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:59:43.613277 32540 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 13:59:43.616096 32540 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } } }
I20240501 13:59:43.617075 32203 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } } }
I20240501 13:59:43.617292 32353 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } } }
I20240501 13:59:43.621357 32101 catalog_manager.cc:5484] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb reported cstate change: config changed from index 2 to 3, 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f191e94b435c4186b6048e9c87fbdfeb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 13:59:43.623485 32490 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 82f81984814f44eea5eafef3ebdf7156 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "2a8d3013d7cc4ee7a904f9e5a4334708" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 43179 } attrs { replace: true } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } } }
I20240501 13:59:43.636955 32278 consensus_queue.cc:237] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } }
I20240501 13:59:43.638815 32490 raft_consensus.cc:1261] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 FOLLOWER]: Refusing update from remote peer f191e94b435c4186b6048e9c87fbdfeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:43.639269 32203 raft_consensus.cc:1261] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Refusing update from remote peer f191e94b435c4186b6048e9c87fbdfeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 13:59:43.639356 32542 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:43.639868 32542 consensus_queue.cc:1035] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 13:59:43.642045 32540 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } } }
I20240501 13:59:43.645287 32203 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } } }
I20240501 13:59:43.645282 32490 raft_consensus.cc:2932] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } } }
I20240501 13:59:43.645869 32089 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c02ea2c5d074862a9286ec08fcb2cb4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 13:59:43.646452 32100 catalog_manager.cc:5484] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb reported cstate change: config changed from index 3 to 4, VOTER 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195) evicted. New cstate: current_term: 1 leader_uuid: "f191e94b435c4186b6048e9c87fbdfeb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4482b523ee4441bd82289235f259aa65" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f191e94b435c4186b6048e9c87fbdfeb" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f81984814f44eea5eafef3ebdf7156" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 36515 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 13:59:43.651508 32343 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c02ea2c5d074862a9286ec08fcb2cb4 with delete_type TABLET_DATA_TOMBSTONED (TS 2a8d3013d7cc4ee7a904f9e5a4334708 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34656
I20240501 13:59:43.652985 32554 tablet_replica.cc:330] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: stopping tablet replica
I20240501 13:59:43.653170 32554 raft_consensus.cc:2229] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:43.653566 32554 raft_consensus.cc:2258] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:43.654367 32554 ts_tablet_manager.cc:1875] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 13:59:43.659502 32554 ts_tablet_manager.cc:1888] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 13:59:43.659651 32554 log.cc:1196] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 2a8d3013d7cc4ee7a904f9e5a4334708: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714571949891039-30475-0/minicluster-data/ts-2-root/wals/8c02ea2c5d074862a9286ec08fcb2cb4
I20240501 13:59:43.660396 32086 catalog_manager.cc:4830] TS 2a8d3013d7cc4ee7a904f9e5a4334708 (127.29.194.195:43179): tablet 8c02ea2c5d074862a9286ec08fcb2cb4 (table test-workload [id=01b8881933dd411eb270f6e3c79c76c0]) successfully deleted
I20240501 13:59:45.081125 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:45.081797 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:45.081935 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:47.082419 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:47.083076 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:47.083220 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:49.083619 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:49.084178 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:49.084293 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:51.084766 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:51.085453 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:51.085565 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:53.085937 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:53.086510 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:53.086616 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:55.086992 32155 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 13:59:55.087661 32155 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 13:59:55.087777 32155 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 13:59:55.680783 30475 tablet_server.cc:178] TabletServer@127.29.194.193:0 shutting down...
I20240501 13:59:55.692867 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:55.693117 30475 tablet_replica.cc:330] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65: stopping tablet replica
I20240501 13:59:55.693311 30475 raft_consensus.cc:2229] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:55.693567 30475 raft_consensus.cc:2258] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 4482b523ee4441bd82289235f259aa65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:55.694131 30475 tablet_replica.cc:330] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65: stopping tablet replica
I20240501 13:59:55.694234 30475 raft_consensus.cc:2229] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:55.694346 30475 raft_consensus.cc:2258] T 43772016e716482385d60c4b01a919e5 P 4482b523ee4441bd82289235f259aa65 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:55.710417 30475 tablet_server.cc:194] TabletServer@127.29.194.193:0 shutdown complete.
I20240501 13:59:55.713521 30475 tablet_server.cc:178] TabletServer@127.29.194.194:0 shutting down...
I20240501 13:59:55.722031 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:55.722290 30475 tablet_replica.cc:330] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb: stopping tablet replica
I20240501 13:59:55.722460 30475 raft_consensus.cc:2229] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:55.722633 30475 raft_consensus.cc:2258] T 43772016e716482385d60c4b01a919e5 P f191e94b435c4186b6048e9c87fbdfeb [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:55.723163 30475 tablet_replica.cc:330] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb: stopping tablet replica
I20240501 13:59:55.723310 30475 raft_consensus.cc:2229] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:55.723598 30475 raft_consensus.cc:2258] T 8c02ea2c5d074862a9286ec08fcb2cb4 P f191e94b435c4186b6048e9c87fbdfeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:55.737287 30475 tablet_server.cc:194] TabletServer@127.29.194.194:0 shutdown complete.
I20240501 13:59:55.740274 30475 tablet_server.cc:178] TabletServer@127.29.194.195:0 shutting down...
I20240501 13:59:55.748998 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:55.749279 30475 tablet_replica.cc:330] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708: stopping tablet replica
I20240501 13:59:55.749423 30475 raft_consensus.cc:2229] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 2 LEADER]: Raft consensus shutting down.
I20240501 13:59:55.749727 30475 raft_consensus.cc:2258] T 43772016e716482385d60c4b01a919e5 P 2a8d3013d7cc4ee7a904f9e5a4334708 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:55.764075 30475 tablet_server.cc:194] TabletServer@127.29.194.195:0 shutdown complete.
I20240501 13:59:55.767043 30475 tablet_server.cc:178] TabletServer@127.29.194.196:0 shutting down...
I20240501 13:59:55.776062 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 13:59:55.776314 30475 tablet_replica.cc:330] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156: stopping tablet replica
I20240501 13:59:55.776473 30475 raft_consensus.cc:2229] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 13:59:55.776719 30475 raft_consensus.cc:2258] T 8c02ea2c5d074862a9286ec08fcb2cb4 P 82f81984814f44eea5eafef3ebdf7156 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:55.790401 30475 tablet_server.cc:194] TabletServer@127.29.194.196:0 shutdown complete.
I20240501 13:59:55.793134 30475 master.cc:537] Master@127.29.194.254:43207 shutting down...
I20240501 13:59:55.806685 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 1 LEADER]: Raft consensus shutting down.
I20240501 13:59:55.806892 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 13:59:55.806984 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2edc832e9c7d4909bbf078254a0fa970: stopping tablet replica
I20240501 13:59:55.819962 30475 master.cc:552] Master@127.29.194.254:43207 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26840 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240501 13:59:55.831429 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:37863
I20240501 13:59:55.831871 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 13:59:55.834715 32558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:55.835412 32559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 13:59:55.836105 32561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 13:59:55.836794 30475 server_base.cc:1034] running on GCE node
I20240501 13:59:55.837002 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 13:59:55.837061 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 13:59:55.837105 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714571995837101 us; error 0 us; skew 500 ppm
I20240501 13:59:55.837265 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 13:59:55.852326 30475 webserver.cc:460] Webserver started at http://127.29.194.254:34457/ using document root <none> and password file <none>
I20240501 13:59:55.852574 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 13:59:55.852635 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 13:59:55.852777 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 13:59:55.853363 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "a53d2c65561145ca843d9efe2f870986"
format_stamp: "Formatted at 2024-05-01 13:59:55 on dist-test-slave-kwk7"
I20240501 13:59:55.855769 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 13:59:55.857182 32566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 13:59:55.857514 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 13:59:55.857604 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "a53d2c65561145ca843d9efe2f870986"
format_stamp: "Formatted at 2024-05-01 13:59:55 on dist-test-slave-kwk7"
I20240501 13:59:55.857712 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 13:59:55.876456 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 13:59:55.877000 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 13:59:55.886912 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:37863
I20240501 13:59:55.886942 32617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:37863 every 8 connection(s)
I20240501 13:59:55.888031 32618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 13:59:55.891858 32618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986: Bootstrap starting.
I20240501 13:59:55.892853 32618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986: Neither blocks nor log segments found. Creating new log.
I20240501 13:59:55.894085 32618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986: No bootstrap required, opened a new log
I20240501 13:59:55.894620 32618 raft_consensus.cc:348] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53d2c65561145ca843d9efe2f870986" member_type: VOTER }
I20240501 13:59:55.894732 32618 raft_consensus.cc:374] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 13:59:55.894769 32618 raft_consensus.cc:729] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53d2c65561145ca843d9efe2f870986, State: Initialized, Role: FOLLOWER
I20240501 13:59:55.894934 32618 consensus_queue.cc:260] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53d2c65561145ca843d9efe2f870986" member_type: VOTER }
I20240501 13:59:55.895064 32618 raft_consensus.cc:388] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 13:59:55.895107 32618 raft_consensus.cc:482] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 13:59:55.895156 32618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 0 FOLLOWER]: Advancing to term 1
I20240501 13:59:55.896775 32618 raft_consensus.cc:504] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53d2c65561145ca843d9efe2f870986" member_type: VOTER }
I20240501 13:59:55.896950 32618 leader_election.cc:302] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a53d2c65561145ca843d9efe2f870986; no voters:
I20240501 13:59:55.897297 32618 leader_election.cc:288] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 13:59:55.897384 32621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 13:59:55.897871 32621 raft_consensus.cc:686] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 1 LEADER]: Becoming Leader. State: Replica: a53d2c65561145ca843d9efe2f870986, State: Running, Role: LEADER
I20240501 13:59:55.898106 32618 sys_catalog.cc:564] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 13:59:55.898073 32621 consensus_queue.cc:237] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53d2c65561145ca843d9efe2f870986" member_type: VOTER }
I20240501 13:59:55.898792 32623 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53d2c65561145ca843d9efe2f870986. Latest consensus state: current_term: 1 leader_uuid: "a53d2c65561145ca843d9efe2f870986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53d2c65561145ca843d9efe2f870986" member_type: VOTER } }
I20240501 13:59:55.899041 32623 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:55.899127 32622 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a53d2c65561145ca843d9efe2f870986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53d2c65561145ca843d9efe2f870986" member_type: VOTER } }
I20240501 13:59:55.899314 32622 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [sys.catalog]: This master's current role is: LEADER
I20240501 13:59:55.900203 32626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 13:59:55.902065 32626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 13:59:55.903859 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 13:59:55.905263 32626 catalog_manager.cc:1347] Generated new cluster ID: d222b4a4075046c09bed646900355f34
I20240501 13:59:55.905344 32626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 13:59:55.916749 32626 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 13:59:55.917476 32626 catalog_manager.cc:1504] Loading token signing keys...
I20240501 13:59:55.923859 32626 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986: Generated new TSK 0
I20240501 13:59:55.924077 32626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 13:59:55.936097 30475 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 9.0732e-05s
I20240501 14:00:08.017880 30475 master.cc:537] Master@127.29.194.254:37863 shutting down...
I20240501 14:00:08.030529 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:08.030778 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:08.030887 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P a53d2c65561145ca843d9efe2f870986: stopping tablet replica
I20240501 14:00:08.044677 30475 master.cc:552] Master@127.29.194.254:37863 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12221 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240501 14:00:08.053074 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:46453
I20240501 14:00:08.053553 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:08.056895 32640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.057662 32641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.058117 32643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:08.059399 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:08.060842 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:08.060914 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:08.060990 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572008060965 us; error 0 us; skew 500 ppm
I20240501 14:00:08.061157 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:08.067577 30475 webserver.cc:460] Webserver started at http://127.29.194.254:46785/ using document root <none> and password file <none>
I20240501 14:00:08.067813 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:08.067894 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:08.068038 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:08.068656 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "ca85fe28423544c3b9fd5e16707b046b"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-kwk7"
I20240501 14:00:08.071277 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:08.072721 32648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.073096 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:08.073190 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "ca85fe28423544c3b9fd5e16707b046b"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-kwk7"
I20240501 14:00:08.073290 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:08.082481 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:08.082909 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:08.092924 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:46453
I20240501 14:00:08.092942 32699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:46453 every 8 connection(s)
I20240501 14:00:08.094084 32700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:08.098083 32700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b: Bootstrap starting.
I20240501 14:00:08.099115 32700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:08.100503 32700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b: No bootstrap required, opened a new log
I20240501 14:00:08.101069 32700 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca85fe28423544c3b9fd5e16707b046b" member_type: VOTER }
I20240501 14:00:08.101199 32700 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:08.101275 32700 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca85fe28423544c3b9fd5e16707b046b, State: Initialized, Role: FOLLOWER
I20240501 14:00:08.101477 32700 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca85fe28423544c3b9fd5e16707b046b" member_type: VOTER }
I20240501 14:00:08.101603 32700 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:00:08.101680 32700 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:00:08.101771 32700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:08.103438 32700 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca85fe28423544c3b9fd5e16707b046b" member_type: VOTER }
I20240501 14:00:08.103628 32700 leader_election.cc:302] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca85fe28423544c3b9fd5e16707b046b; no voters:
I20240501 14:00:08.103983 32700 leader_election.cc:288] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:00:08.104075 32703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:08.104598 32703 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 1 LEADER]: Becoming Leader. State: Replica: ca85fe28423544c3b9fd5e16707b046b, State: Running, Role: LEADER
I20240501 14:00:08.104866 32703 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca85fe28423544c3b9fd5e16707b046b" member_type: VOTER }
I20240501 14:00:08.104892 32700 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:00:08.105724 32704 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca85fe28423544c3b9fd5e16707b046b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca85fe28423544c3b9fd5e16707b046b" member_type: VOTER } }
I20240501 14:00:08.105791 32705 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca85fe28423544c3b9fd5e16707b046b. Latest consensus state: current_term: 1 leader_uuid: "ca85fe28423544c3b9fd5e16707b046b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca85fe28423544c3b9fd5e16707b046b" member_type: VOTER } }
I20240501 14:00:08.105979 32704 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:08.106081 32705 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:08.106981 32708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:00:08.108204 32708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:00:08.110105 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:00:08.110925 32708 catalog_manager.cc:1347] Generated new cluster ID: 0613908fc00446a3b58ee72ba78bc69d
I20240501 14:00:08.111013 32708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:00:08.129264 32708 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:00:08.130087 32708 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:00:08.141750 32708 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b: Generated new TSK 0
I20240501 14:00:08.142000 32708 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:00:08.175141 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:08.178401 32721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.179061 32722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.179682 32724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:08.180306 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:08.180539 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:08.180595 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:08.180636 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572008180622 us; error 0 us; skew 500 ppm
I20240501 14:00:08.180814 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:08.181998 30475 webserver.cc:460] Webserver started at http://127.29.194.193:44753/ using document root <none> and password file <none>
I20240501 14:00:08.182195 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:08.182264 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:08.182366 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:08.182955 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "de4c8d34ec834c5d8da6c486d6e5847d"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-kwk7"
I20240501 14:00:08.185359 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:08.186750 32729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.187124 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:08.187240 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "de4c8d34ec834c5d8da6c486d6e5847d"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-kwk7"
I20240501 14:00:08.187361 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:08.200793 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:08.201326 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:08.201854 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:08.202672 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:08.202747 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.202855 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:08.202910 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.215616 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:34145
I20240501 14:00:08.215636 323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:34145 every 8 connection(s)
I20240501 14:00:08.217262 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:08.221086 328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.221741 329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:08.222321 30475 server_base.cc:1034] running on GCE node
W20240501 14:00:08.222858 331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:08.223337 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:08.223436 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:08.223496 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572008223473 us; error 0 us; skew 500 ppm
I20240501 14:00:08.223685 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:08.225168 324 heartbeater.cc:347] Connected to a master server at 127.29.194.254:46453
I20240501 14:00:08.225200 30475 webserver.cc:460] Webserver started at http://127.29.194.194:39167/ using document root <none> and password file <none>
I20240501 14:00:08.225332 324 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:08.225473 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:08.225566 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:08.225718 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:08.225754 324 heartbeater.cc:511] Master 127.29.194.254:46453 requested a full tablet report, sending...
I20240501 14:00:08.226534 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "cdc75fc9024e46679d38a6ac80544688"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-kwk7"
I20240501 14:00:08.226749 32665 ts_manager.cc:194] Registered new tserver with Master: de4c8d34ec834c5d8da6c486d6e5847d (127.29.194.193:34145)
I20240501 14:00:08.227761 32665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42328
I20240501 14:00:08.229187 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20240501 14:00:08.230659 336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.231009 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 14:00:08.231112 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "cdc75fc9024e46679d38a6ac80544688"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-kwk7"
I20240501 14:00:08.231222 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:08.249488 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:08.250046 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:08.250577 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:08.251452 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:08.251529 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.251636 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:08.251698 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.264433 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:42583
I20240501 14:00:08.264475 398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:42583 every 8 connection(s)
I20240501 14:00:08.266122 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:08.269809 402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.270540 403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:08.271184 405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:08.271849 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:08.272183 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:08.272277 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:08.272326 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572008272308 us; error 0 us; skew 500 ppm
I20240501 14:00:08.272481 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:08.272778 399 heartbeater.cc:347] Connected to a master server at 127.29.194.254:46453
I20240501 14:00:08.272905 399 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:08.273355 399 heartbeater.cc:511] Master 127.29.194.254:46453 requested a full tablet report, sending...
I20240501 14:00:08.273921 30475 webserver.cc:460] Webserver started at http://127.29.194.195:34191/ using document root <none> and password file <none>
I20240501 14:00:08.274158 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:08.274245 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:08.274379 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:08.274355 32665 ts_manager.cc:194] Registered new tserver with Master: cdc75fc9024e46679d38a6ac80544688 (127.29.194.194:42583)
I20240501 14:00:08.275192 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "1dbaa982da90424fa3424d32e2b2127f"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-kwk7"
I20240501 14:00:08.275467 32665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42332
I20240501 14:00:08.277938 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:08.279466 410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.279816 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 14:00:08.279911 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "1dbaa982da90424fa3424d32e2b2127f"
format_stamp: "Formatted at 2024-05-01 14:00:08 on dist-test-slave-kwk7"
I20240501 14:00:08.280032 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:08.313436 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:08.314042 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:08.314641 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:08.315470 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:08.315548 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.315650 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:08.315714 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:08.327951 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:45371
I20240501 14:00:08.327986 472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:45371 every 8 connection(s)
I20240501 14:00:08.334697 473 heartbeater.cc:347] Connected to a master server at 127.29.194.254:46453
I20240501 14:00:08.334844 473 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:08.335327 473 heartbeater.cc:511] Master 127.29.194.254:46453 requested a full tablet report, sending...
I20240501 14:00:08.336158 32665 ts_manager.cc:194] Registered new tserver with Master: 1dbaa982da90424fa3424d32e2b2127f (127.29.194.195:45371)
I20240501 14:00:08.336781 30475 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007926296s
I20240501 14:00:08.337075 32665 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42336
I20240501 14:00:09.229100 324 heartbeater.cc:503] Master 127.29.194.254:46453 was elected leader, sending a full tablet report...
I20240501 14:00:09.276964 399 heartbeater.cc:503] Master 127.29.194.254:46453 was elected leader, sending a full tablet report...
I20240501 14:00:09.338465 473 heartbeater.cc:503] Master 127.29.194.254:46453 was elected leader, sending a full tablet report...
I20240501 14:00:10.109786 32718 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:12.110191 32718 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:14.110620 32718 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:16.111078 32718 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:18.111516 32718 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:20.111905 32718 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:20.418774 30475 tablet_server.cc:178] TabletServer@127.29.194.193:0 shutting down...
I20240501 14:00:20.433550 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:20.447379 30475 tablet_server.cc:194] TabletServer@127.29.194.193:0 shutdown complete.
I20240501 14:00:20.449875 30475 tablet_server.cc:178] TabletServer@127.29.194.194:0 shutting down...
I20240501 14:00:20.457728 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:20.470132 30475 tablet_server.cc:194] TabletServer@127.29.194.194:0 shutdown complete.
I20240501 14:00:20.472623 30475 tablet_server.cc:178] TabletServer@127.29.194.195:0 shutting down...
I20240501 14:00:20.480897 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:20.493577 30475 tablet_server.cc:194] TabletServer@127.29.194.195:0 shutdown complete.
I20240501 14:00:20.495965 30475 master.cc:537] Master@127.29.194.254:46453 shutting down...
I20240501 14:00:20.504230 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:20.504431 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:20.504537 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P ca85fe28423544c3b9fd5e16707b046b: stopping tablet replica
I20240501 14:00:20.517573 30475 master.cc:552] Master@127.29.194.254:46453 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12473 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240501 14:00:20.526589 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:37557
I20240501 14:00:20.527088 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:20.530136 477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.530848 478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.531437 480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:20.532156 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:20.532366 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:20.532420 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:20.532460 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572020532446 us; error 0 us; skew 500 ppm
I20240501 14:00:20.532595 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:20.533833 30475 webserver.cc:460] Webserver started at http://127.29.194.254:39225/ using document root <none> and password file <none>
I20240501 14:00:20.534026 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:20.534087 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:20.534226 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:20.534837 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "d10fc296e1c8483eadb5348c6b859b2c"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.537259 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:20.538698 485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.539058 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:20.539160 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "d10fc296e1c8483eadb5348c6b859b2c"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.539279 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:20.547711 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:20.548209 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:20.558506 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:37557
I20240501 14:00:20.558528 536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:37557 every 8 connection(s)
I20240501 14:00:20.559636 537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:20.563480 537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c: Bootstrap starting.
I20240501 14:00:20.564570 537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:20.565965 537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c: No bootstrap required, opened a new log
I20240501 14:00:20.566491 537 raft_consensus.cc:348] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10fc296e1c8483eadb5348c6b859b2c" member_type: VOTER }
I20240501 14:00:20.566609 537 raft_consensus.cc:374] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:20.566661 537 raft_consensus.cc:729] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d10fc296e1c8483eadb5348c6b859b2c, State: Initialized, Role: FOLLOWER
I20240501 14:00:20.566857 537 consensus_queue.cc:260] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10fc296e1c8483eadb5348c6b859b2c" member_type: VOTER }
I20240501 14:00:20.566977 537 raft_consensus.cc:388] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:00:20.567039 537 raft_consensus.cc:482] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:00:20.567102 537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:20.568717 537 raft_consensus.cc:504] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10fc296e1c8483eadb5348c6b859b2c" member_type: VOTER }
I20240501 14:00:20.568903 537 leader_election.cc:302] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d10fc296e1c8483eadb5348c6b859b2c; no voters:
I20240501 14:00:20.569226 537 leader_election.cc:288] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:00:20.569341 540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:20.569814 540 raft_consensus.cc:686] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 1 LEADER]: Becoming Leader. State: Replica: d10fc296e1c8483eadb5348c6b859b2c, State: Running, Role: LEADER
I20240501 14:00:20.570060 537 sys_catalog.cc:564] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:00:20.570040 540 consensus_queue.cc:237] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10fc296e1c8483eadb5348c6b859b2c" member_type: VOTER }
I20240501 14:00:20.570822 542 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d10fc296e1c8483eadb5348c6b859b2c. Latest consensus state: current_term: 1 leader_uuid: "d10fc296e1c8483eadb5348c6b859b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10fc296e1c8483eadb5348c6b859b2c" member_type: VOTER } }
I20240501 14:00:20.571019 542 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:20.573350 545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:00:20.573383 541 sys_catalog.cc:455] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d10fc296e1c8483eadb5348c6b859b2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d10fc296e1c8483eadb5348c6b859b2c" member_type: VOTER } }
I20240501 14:00:20.573684 541 sys_catalog.cc:458] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:20.574710 545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:00:20.575438 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:00:20.577454 545 catalog_manager.cc:1347] Generated new cluster ID: 01429b9e86cd42c1803a9faf18a73225
I20240501 14:00:20.577560 545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:00:20.584496 545 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:00:20.585249 545 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:00:20.595309 545 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c: Generated new TSK 0
I20240501 14:00:20.595539 545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:00:20.608318 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:20.611667 558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.612321 559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.612906 561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:20.613584 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:20.613814 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:20.613893 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:20.613934 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572020613919 us; error 0 us; skew 500 ppm
I20240501 14:00:20.614041 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:20.615247 30475 webserver.cc:460] Webserver started at http://127.29.194.193:45013/ using document root <none> and password file <none>
I20240501 14:00:20.615450 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:20.615530 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:20.615639 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:20.616211 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "0cfc40f4513048418e30774ccd5242d1"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.618695 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:20.620121 566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.620456 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 14:00:20.620558 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "0cfc40f4513048418e30774ccd5242d1"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.620658 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:20.631294 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:20.631772 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:20.632298 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:20.633148 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:20.633208 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.633296 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:20.633340 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.646703 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:33799
I20240501 14:00:20.646708 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:33799 every 8 connection(s)
I20240501 14:00:20.648205 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:20.651974 633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.652611 634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.653318 636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:20.653997 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:20.654295 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:20.654378 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:20.654461 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572020654421 us; error 0 us; skew 500 ppm
I20240501 14:00:20.654647 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:20.656209 30475 webserver.cc:460] Webserver started at http://127.29.194.194:34923/ using document root <none> and password file <none>
I20240501 14:00:20.656481 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:20.656598 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:20.656737 629 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37557
I20240501 14:00:20.656778 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:20.656875 629 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:20.657258 629 heartbeater.cc:511] Master 127.29.194.254:37557 requested a full tablet report, sending...
I20240501 14:00:20.657532 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "dc4229093f50479f8689f9d4c2705a51"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.658257 502 ts_manager.cc:194] Registered new tserver with Master: 0cfc40f4513048418e30774ccd5242d1 (127.29.194.193:33799)
I20240501 14:00:20.659487 502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52916
I20240501 14:00:20.660298 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:20.661814 641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.662099 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:20.662206 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "dc4229093f50479f8689f9d4c2705a51"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.662317 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:20.685199 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:20.685794 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:20.686347 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:20.687127 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:20.687196 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.687299 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:20.687355 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.700585 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:35355
I20240501 14:00:20.700629 703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:35355 every 8 connection(s)
I20240501 14:00:20.702086 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:20.705806 707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.706394 708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.707981 710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:20.708786 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:20.709118 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 14:00:20.709158 704 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37557
W20240501 14:00:20.709218 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:20.709285 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572020709255 us; error 0 us; skew 500 ppm
I20240501 14:00:20.709288 704 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:20.709465 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:20.709781 704 heartbeater.cc:511] Master 127.29.194.254:37557 requested a full tablet report, sending...
I20240501 14:00:20.710774 502 ts_manager.cc:194] Registered new tserver with Master: dc4229093f50479f8689f9d4c2705a51 (127.29.194.194:35355)
I20240501 14:00:20.710774 30475 webserver.cc:460] Webserver started at http://127.29.194.195:40103/ using document root <none> and password file <none>
I20240501 14:00:20.711074 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:20.711169 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:20.711311 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:20.711836 502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52928
I20240501 14:00:20.712136 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "958b53bcda80422e88bb3eca91e6fce0"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.714874 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:20.716333 715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.716670 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:20.716810 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "958b53bcda80422e88bb3eca91e6fce0"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.716928 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:20.733615 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:20.734155 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:20.734670 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:20.735457 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:20.735538 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.735638 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:20.735693 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.747797 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:44911
I20240501 14:00:20.747833 777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:44911 every 8 connection(s)
I20240501 14:00:20.749284 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:20.752885 781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.753612 782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:20.754565 784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:20.756013 778 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37557
I20240501 14:00:20.756137 778 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:20.756251 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:20.756515 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 14:00:20.756520 778 heartbeater.cc:511] Master 127.29.194.254:37557 requested a full tablet report, sending...
I20240501 14:00:20.757447 502 ts_manager.cc:194] Registered new tserver with Master: 958b53bcda80422e88bb3eca91e6fce0 (127.29.194.195:44911)
I20240501 14:00:20.758175 502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52940
W20240501 14:00:20.766284 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:20.766342 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572020766338 us; error 0 us; skew 500 ppm
I20240501 14:00:20.766465 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:20.767570 30475 webserver.cc:460] Webserver started at http://127.29.194.196:37363/ using document root <none> and password file <none>
I20240501 14:00:20.767750 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:20.767812 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:20.767921 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:20.768451 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-3-root/instance:
uuid: "2a1ad05e67bc4b5290ac59b3f70ff276"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.770572 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:20.771937 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.772296 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 14:00:20.772404 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-3-root
uuid: "2a1ad05e67bc4b5290ac59b3f70ff276"
format_stamp: "Formatted at 2024-05-01 14:00:20 on dist-test-slave-kwk7"
I20240501 14:00:20.772505 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714571949891039-30475-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:20.783594 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:20.783982 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:20.784389 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:20.785133 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:20.785181 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.785233 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:20.785277 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:20.795305 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.196:34503
I20240501 14:00:20.795331 851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.196:34503 every 8 connection(s)
I20240501 14:00:20.800997 852 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37557
I20240501 14:00:20.801111 852 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:20.801384 852 heartbeater.cc:511] Master 127.29.194.254:37557 requested a full tablet report, sending...
I20240501 14:00:20.802132 502 ts_manager.cc:194] Registered new tserver with Master: 2a1ad05e67bc4b5290ac59b3f70ff276 (127.29.194.196:34503)
I20240501 14:00:20.802788 502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52952
I20240501 14:00:20.803069 30475 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.006971991s
I20240501 14:00:21.660863 629 heartbeater.cc:503] Master 127.29.194.254:37557 was elected leader, sending a full tablet report...
I20240501 14:00:21.713311 704 heartbeater.cc:503] Master 127.29.194.254:37557 was elected leader, sending a full tablet report...
I20240501 14:00:21.759384 778 heartbeater.cc:503] Master 127.29.194.254:37557 was elected leader, sending a full tablet report...
I20240501 14:00:21.803839 852 heartbeater.cc:503] Master 127.29.194.254:37557 was elected leader, sending a full tablet report...
I20240501 14:00:21.836228 30475 test_util.cc:268] Using random seed: -397563472
I20240501 14:00:21.846328 502 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52978:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 14:00:21.862664 817 tablet_service.cc:1467] Processing CreateTablet for tablet ee0fa95a2c5e4706bb66bb2530f30f14 (DEFAULT_TABLE table=test-workload [id=e114e71927074f158a0a8476eedb4aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.863114 817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0fa95a2c5e4706bb66bb2530f30f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.867400 743 tablet_service.cc:1467] Processing CreateTablet for tablet ee0fa95a2c5e4706bb66bb2530f30f14 (DEFAULT_TABLE table=test-workload [id=e114e71927074f158a0a8476eedb4aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.867822 743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0fa95a2c5e4706bb66bb2530f30f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.869217 873 tablet_bootstrap.cc:492] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276: Bootstrap starting.
I20240501 14:00:21.870779 873 tablet_bootstrap.cc:654] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.872462 873 tablet_bootstrap.cc:492] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276: No bootstrap required, opened a new log
I20240501 14:00:21.872635 873 ts_tablet_manager.cc:1367] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:21.872905 594 tablet_service.cc:1467] Processing CreateTablet for tablet ee0fa95a2c5e4706bb66bb2530f30f14 (DEFAULT_TABLE table=test-workload [id=e114e71927074f158a0a8476eedb4aa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:21.873294 594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee0fa95a2c5e4706bb66bb2530f30f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:21.873454 873 raft_consensus.cc:348] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:21.873613 873 raft_consensus.cc:374] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.873674 873 raft_consensus.cc:729] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a1ad05e67bc4b5290ac59b3f70ff276, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.873874 873 consensus_queue.cc:260] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:21.874589 873 ts_tablet_manager.cc:1398] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:21.877830 876 tablet_bootstrap.cc:492] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0: Bootstrap starting.
I20240501 14:00:21.878401 877 tablet_bootstrap.cc:492] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1: Bootstrap starting.
I20240501 14:00:21.879276 876 tablet_bootstrap.cc:654] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.879703 877 tablet_bootstrap.cc:654] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:21.880810 876 tablet_bootstrap.cc:492] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0: No bootstrap required, opened a new log
I20240501 14:00:21.880936 876 ts_tablet_manager.cc:1367] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:21.881170 877 tablet_bootstrap.cc:492] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1: No bootstrap required, opened a new log
I20240501 14:00:21.881278 877 ts_tablet_manager.cc:1367] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:21.881649 876 raft_consensus.cc:348] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:21.881826 876 raft_consensus.cc:374] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.881894 876 raft_consensus.cc:729] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 958b53bcda80422e88bb3eca91e6fce0, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.881860 877 raft_consensus.cc:348] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:21.882009 877 raft_consensus.cc:374] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:21.882050 877 raft_consensus.cc:729] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfc40f4513048418e30774ccd5242d1, State: Initialized, Role: FOLLOWER
I20240501 14:00:21.882084 876 consensus_queue.cc:260] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:21.882230 877 consensus_queue.cc:260] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:21.882831 876 ts_tablet_manager.cc:1398] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:21.883229 877 ts_tablet_manager.cc:1398] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240501 14:00:21.898802 630 tablet.cc:2369] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:00:22.035140 880 raft_consensus.cc:482] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:22.035358 880 raft_consensus.cc:504] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:22.036329 880 leader_election.cc:288] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a1ad05e67bc4b5290ac59b3f70ff276 (127.29.194.196:34503), 0cfc40f4513048418e30774ccd5242d1 (127.29.194.193:33799)
I20240501 14:00:22.042285 827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0fa95a2c5e4706bb66bb2530f30f14" candidate_uuid: "958b53bcda80422e88bb3eca91e6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" is_pre_election: true
I20240501 14:00:22.042538 827 raft_consensus.cc:2454] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 958b53bcda80422e88bb3eca91e6fce0 in term 0.
I20240501 14:00:22.042580 604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0fa95a2c5e4706bb66bb2530f30f14" candidate_uuid: "958b53bcda80422e88bb3eca91e6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfc40f4513048418e30774ccd5242d1" is_pre_election: true
I20240501 14:00:22.042793 604 raft_consensus.cc:2454] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 958b53bcda80422e88bb3eca91e6fce0 in term 0.
I20240501 14:00:22.042945 717 leader_election.cc:302] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a1ad05e67bc4b5290ac59b3f70ff276, 958b53bcda80422e88bb3eca91e6fce0; no voters:
I20240501 14:00:22.043242 880 raft_consensus.cc:2781] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:22.043319 880 raft_consensus.cc:482] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:22.043397 880 raft_consensus.cc:3037] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.045238 880 raft_consensus.cc:504] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:22.045684 880 leader_election.cc:288] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [CANDIDATE]: Term 1 election: Requested vote from peers 2a1ad05e67bc4b5290ac59b3f70ff276 (127.29.194.196:34503), 0cfc40f4513048418e30774ccd5242d1 (127.29.194.193:33799)
I20240501 14:00:22.046061 827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0fa95a2c5e4706bb66bb2530f30f14" candidate_uuid: "958b53bcda80422e88bb3eca91e6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276"
I20240501 14:00:22.046118 604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee0fa95a2c5e4706bb66bb2530f30f14" candidate_uuid: "958b53bcda80422e88bb3eca91e6fce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfc40f4513048418e30774ccd5242d1"
I20240501 14:00:22.046212 827 raft_consensus.cc:3037] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.046241 604 raft_consensus.cc:3037] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:22.047847 604 raft_consensus.cc:2454] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 958b53bcda80422e88bb3eca91e6fce0 in term 1.
I20240501 14:00:22.048027 827 raft_consensus.cc:2454] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 958b53bcda80422e88bb3eca91e6fce0 in term 1.
I20240501 14:00:22.048162 717 leader_election.cc:302] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfc40f4513048418e30774ccd5242d1, 958b53bcda80422e88bb3eca91e6fce0; no voters:
I20240501 14:00:22.048400 880 raft_consensus.cc:2781] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:22.048821 880 raft_consensus.cc:686] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 1 LEADER]: Becoming Leader. State: Replica: 958b53bcda80422e88bb3eca91e6fce0, State: Running, Role: LEADER
I20240501 14:00:22.049237 880 consensus_queue.cc:237] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } }
I20240501 14:00:22.051493 500 catalog_manager.cc:5484] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 958b53bcda80422e88bb3eca91e6fce0 (127.29.194.195). New cstate: current_term: 1 leader_uuid: "958b53bcda80422e88bb3eca91e6fce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "958b53bcda80422e88bb3eca91e6fce0" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 44911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:22.462515 880 consensus_queue.cc:1035] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a1ad05e67bc4b5290ac59b3f70ff276" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 34503 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.467352 884 consensus_queue.cc:1035] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfc40f4513048418e30774ccd5242d1" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 33799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:22.575362 555 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:22.575731 555 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:22.575819 555 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:24.576246 555 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:24.576773 555 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:24.576874 555 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:26.577210 555 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:26.577626 555 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:26.577719 555 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:28.578073 555 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:28.578511 555 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:28.578629 555 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:30.578981 555 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:30.579440 555 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:30.579555 555 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:32.580047 555 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:32.580466 555 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:32.580569 555 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:33.143414 30475 tablet_server.cc:178] TabletServer@127.29.194.193:0 shutting down...
I20240501 14:00:33.155700 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:33.155972 30475 tablet_replica.cc:330] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1: stopping tablet replica
I20240501 14:00:33.156155 30475 raft_consensus.cc:2229] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:33.156363 30475 raft_consensus.cc:2258] T ee0fa95a2c5e4706bb66bb2530f30f14 P 0cfc40f4513048418e30774ccd5242d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:33.169939 30475 tablet_server.cc:194] TabletServer@127.29.194.193:0 shutdown complete.
I20240501 14:00:33.172927 30475 tablet_server.cc:178] TabletServer@127.29.194.194:0 shutting down...
I20240501 14:00:33.180868 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:33.193332 30475 tablet_server.cc:194] TabletServer@127.29.194.194:0 shutdown complete.
I20240501 14:00:33.195811 30475 tablet_server.cc:178] TabletServer@127.29.194.195:0 shutting down...
I20240501 14:00:33.203502 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:33.203730 30475 tablet_replica.cc:330] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0: stopping tablet replica
I20240501 14:00:33.203924 30475 raft_consensus.cc:2229] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:33.204217 30475 raft_consensus.cc:2258] T ee0fa95a2c5e4706bb66bb2530f30f14 P 958b53bcda80422e88bb3eca91e6fce0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:33.217383 30475 tablet_server.cc:194] TabletServer@127.29.194.195:0 shutdown complete.
I20240501 14:00:33.219834 30475 tablet_server.cc:178] TabletServer@127.29.194.196:0 shutting down...
I20240501 14:00:33.227437 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:33.227615 30475 tablet_replica.cc:330] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276: stopping tablet replica
I20240501 14:00:33.227780 30475 raft_consensus.cc:2229] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:33.227964 30475 raft_consensus.cc:2258] T ee0fa95a2c5e4706bb66bb2530f30f14 P 2a1ad05e67bc4b5290ac59b3f70ff276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:33.240928 30475 tablet_server.cc:194] TabletServer@127.29.194.196:0 shutdown complete.
I20240501 14:00:33.243602 30475 master.cc:537] Master@127.29.194.254:37557 shutting down...
I20240501 14:00:33.259824 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:33.260066 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:33.260159 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P d10fc296e1c8483eadb5348c6b859b2c: stopping tablet replica
I20240501 14:00:33.273559 30475 master.cc:552] Master@127.29.194.254:37557 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12757 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240501 14:00:33.284477 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:35931
I20240501 14:00:33.285045 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:33.288049 897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.288605 898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.289214 900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:33.289775 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:33.290006 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:33.290062 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:33.290092 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572033290088 us; error 0 us; skew 500 ppm
I20240501 14:00:33.290268 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:33.303700 30475 webserver.cc:460] Webserver started at http://127.29.194.254:44327/ using document root <none> and password file <none>
I20240501 14:00:33.303962 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:33.304028 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:33.304157 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:33.304888 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "26695e81fba9443e99f5758e24996b15"
format_stamp: "Formatted at 2024-05-01 14:00:33 on dist-test-slave-kwk7"
I20240501 14:00:33.307219 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:33.308588 905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.308924 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:33.309013 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "26695e81fba9443e99f5758e24996b15"
format_stamp: "Formatted at 2024-05-01 14:00:33 on dist-test-slave-kwk7"
I20240501 14:00:33.309137 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:33.316676 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:33.317126 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:33.326846 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:35931
I20240501 14:00:33.326856 956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:35931 every 8 connection(s)
I20240501 14:00:33.328104 957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:33.331961 957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15: Bootstrap starting.
I20240501 14:00:33.333048 957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:33.334360 957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15: No bootstrap required, opened a new log
I20240501 14:00:33.334887 957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26695e81fba9443e99f5758e24996b15" member_type: VOTER }
I20240501 14:00:33.335009 957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:33.335060 957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26695e81fba9443e99f5758e24996b15, State: Initialized, Role: FOLLOWER
I20240501 14:00:33.335242 957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26695e81fba9443e99f5758e24996b15" member_type: VOTER }
I20240501 14:00:33.335347 957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:00:33.335403 957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:00:33.335480 957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:33.337105 957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26695e81fba9443e99f5758e24996b15" member_type: VOTER }
I20240501 14:00:33.337275 957 leader_election.cc:302] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26695e81fba9443e99f5758e24996b15; no voters:
I20240501 14:00:33.337600 957 leader_election.cc:288] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:00:33.337694 960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:33.338094 960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 1 LEADER]: Becoming Leader. State: Replica: 26695e81fba9443e99f5758e24996b15, State: Running, Role: LEADER
I20240501 14:00:33.338316 960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26695e81fba9443e99f5758e24996b15" member_type: VOTER }
I20240501 14:00:33.338411 957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:00:33.339023 962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26695e81fba9443e99f5758e24996b15. Latest consensus state: current_term: 1 leader_uuid: "26695e81fba9443e99f5758e24996b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26695e81fba9443e99f5758e24996b15" member_type: VOTER } }
I20240501 14:00:33.339206 962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:33.339095 961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26695e81fba9443e99f5758e24996b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26695e81fba9443e99f5758e24996b15" member_type: VOTER } }
I20240501 14:00:33.339479 961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:33.340234 964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:00:33.341415 964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:00:33.343397 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:00:33.343992 964 catalog_manager.cc:1347] Generated new cluster ID: f5a6ad7682514088b8a24cc11436674a
I20240501 14:00:33.344064 964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:00:33.371775 964 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:00:33.372498 964 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:00:33.378326 964 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15: Generated new TSK 0
I20240501 14:00:33.378515 964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:00:33.408434 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:33.411788 978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.412552 981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.413225 979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:33.413810 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:33.414013 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:33.414067 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:33.414109 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572033414105 us; error 0 us; skew 500 ppm
I20240501 14:00:33.414283 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:33.415495 30475 webserver.cc:460] Webserver started at http://127.29.194.193:38727/ using document root <none> and password file <none>
I20240501 14:00:33.415724 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:33.415791 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:33.415930 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:33.416615 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "3ae7c0b58b514561914cee013441a2b2"
format_stamp: "Formatted at 2024-05-01 14:00:33 on dist-test-slave-kwk7"
I20240501 14:00:33.419028 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:33.420473 986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.420795 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 14:00:33.420892 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "3ae7c0b58b514561914cee013441a2b2"
format_stamp: "Formatted at 2024-05-01 14:00:33 on dist-test-slave-kwk7"
I20240501 14:00:33.420994 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:33.428968 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:33.429395 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:33.429880 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:33.430693 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:33.430764 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.430855 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:33.430891 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.442589 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:41923
I20240501 14:00:33.442615 1048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:41923 every 8 connection(s)
I20240501 14:00:33.444160 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:33.447585 1053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.448282 1054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.448925 1056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:33.449606 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:33.449895 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:33.449972 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:33.450031 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572033450028 us; error 0 us; skew 500 ppm
I20240501 14:00:33.450230 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:33.450955 1049 heartbeater.cc:347] Connected to a master server at 127.29.194.254:35931
I20240501 14:00:33.451094 1049 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:33.451400 1049 heartbeater.cc:511] Master 127.29.194.254:35931 requested a full tablet report, sending...
I20240501 14:00:33.451584 30475 webserver.cc:460] Webserver started at http://127.29.194.194:39391/ using document root <none> and password file <none>
I20240501 14:00:33.451757 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:33.451836 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:33.451956 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:33.452253 922 ts_manager.cc:194] Registered new tserver with Master: 3ae7c0b58b514561914cee013441a2b2 (127.29.194.193:41923)
I20240501 14:00:33.452597 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "dd4606e1b8cb456bb1a53dbaf04546fc"
format_stamp: "Formatted at 2024-05-01 14:00:33 on dist-test-slave-kwk7"
I20240501 14:00:33.453284 922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39484
I20240501 14:00:33.454949 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:33.456312 1061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.456612 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:33.456713 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "dd4606e1b8cb456bb1a53dbaf04546fc"
format_stamp: "Formatted at 2024-05-01 14:00:33 on dist-test-slave-kwk7"
I20240501 14:00:33.456808 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:33.484406 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:33.484957 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:33.485468 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:33.486224 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:33.486277 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.486347 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:33.486402 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.497900 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:33699
I20240501 14:00:33.497931 1123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:33699 every 8 connection(s)
I20240501 14:00:33.499408 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:33.502887 1127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.503645 1128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:33.504262 1130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:33.504906 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:33.505257 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:33.505328 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:33.505373 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572033505369 us; error 0 us; skew 500 ppm
I20240501 14:00:33.505511 1124 heartbeater.cc:347] Connected to a master server at 127.29.194.254:35931
I20240501 14:00:33.505523 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:33.505656 1124 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:33.506009 1124 heartbeater.cc:511] Master 127.29.194.254:35931 requested a full tablet report, sending...
I20240501 14:00:33.506860 30475 webserver.cc:460] Webserver started at http://127.29.194.195:38503/ using document root <none> and password file <none>
I20240501 14:00:33.506889 922 ts_manager.cc:194] Registered new tserver with Master: dd4606e1b8cb456bb1a53dbaf04546fc (127.29.194.194:33699)
I20240501 14:00:33.507078 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:33.507151 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:33.507261 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:33.507828 922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39486
I20240501 14:00:33.507958 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "b31a2332f794495092c03f45258a8ae2"
format_stamp: "Formatted at 2024-05-01 14:00:33 on dist-test-slave-kwk7"
I20240501 14:00:33.510355 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:33.511770 1135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.512141 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 14:00:33.512244 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "b31a2332f794495092c03f45258a8ae2"
format_stamp: "Formatted at 2024-05-01 14:00:33 on dist-test-slave-kwk7"
I20240501 14:00:33.512368 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:33.533098 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:33.533603 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:33.534097 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:33.534927 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:33.534992 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.535096 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:33.535151 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:33.547679 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:46007
I20240501 14:00:33.547714 1197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:46007 every 8 connection(s)
I20240501 14:00:33.553696 1198 heartbeater.cc:347] Connected to a master server at 127.29.194.254:35931
I20240501 14:00:33.553813 1198 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:33.554183 1198 heartbeater.cc:511] Master 127.29.194.254:35931 requested a full tablet report, sending...
I20240501 14:00:33.554955 922 ts_manager.cc:194] Registered new tserver with Master: b31a2332f794495092c03f45258a8ae2 (127.29.194.195:46007)
I20240501 14:00:33.555177 30475 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006709055s
I20240501 14:00:33.555941 922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39496
I20240501 14:00:34.454442 1049 heartbeater.cc:503] Master 127.29.194.254:35931 was elected leader, sending a full tablet report...
I20240501 14:00:34.509002 1124 heartbeater.cc:503] Master 127.29.194.254:35931 was elected leader, sending a full tablet report...
I20240501 14:00:34.557148 1198 heartbeater.cc:503] Master 127.29.194.254:35931 was elected leader, sending a full tablet report...
I20240501 14:00:34.588457 30475 test_util.cc:268] Using random seed: -384811243
I20240501 14:00:34.598552 922 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39508:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:00:34.599035 922 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:00:34.616732 1089 tablet_service.cc:1467] Processing CreateTablet for tablet e18fb36808034d38a643426dbbde0748 (DEFAULT_TABLE table=test-workload [id=22953a2a6f5943bcacb75bc74e3a6683]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.617229 1089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18fb36808034d38a643426dbbde0748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.618548 1014 tablet_service.cc:1467] Processing CreateTablet for tablet e18fb36808034d38a643426dbbde0748 (DEFAULT_TABLE table=test-workload [id=22953a2a6f5943bcacb75bc74e3a6683]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.618868 1014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18fb36808034d38a643426dbbde0748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.619256 1163 tablet_service.cc:1467] Processing CreateTablet for tablet e18fb36808034d38a643426dbbde0748 (DEFAULT_TABLE table=test-workload [id=22953a2a6f5943bcacb75bc74e3a6683]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:00:34.619585 1163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e18fb36808034d38a643426dbbde0748. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:34.623342 1219 tablet_bootstrap.cc:492] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2: Bootstrap starting.
I20240501 14:00:34.624655 1219 tablet_bootstrap.cc:654] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.625113 1220 tablet_bootstrap.cc:492] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc: Bootstrap starting.
I20240501 14:00:34.626067 1221 tablet_bootstrap.cc:492] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2: Bootstrap starting.
I20240501 14:00:34.626724 1220 tablet_bootstrap.cc:654] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.627357 1219 tablet_bootstrap.cc:492] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2: No bootstrap required, opened a new log
I20240501 14:00:34.627447 1221 tablet_bootstrap.cc:654] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:34.627544 1219 ts_tablet_manager.cc:1367] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:34.628326 1219 raft_consensus.cc:348] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.628471 1220 tablet_bootstrap.cc:492] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc: No bootstrap required, opened a new log
I20240501 14:00:34.628551 1219 raft_consensus.cc:374] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.628666 1219 raft_consensus.cc:729] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ae7c0b58b514561914cee013441a2b2, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.628680 1220 ts_tablet_manager.cc:1367] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:00:34.628993 1221 tablet_bootstrap.cc:492] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2: No bootstrap required, opened a new log
I20240501 14:00:34.629149 1221 ts_tablet_manager.cc:1367] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:34.628962 1219 consensus_queue.cc:260] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.629545 1220 raft_consensus.cc:348] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.629854 1220 raft_consensus.cc:374] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.629894 1221 raft_consensus.cc:348] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.630007 1220 raft_consensus.cc:729] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd4606e1b8cb456bb1a53dbaf04546fc, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.630332 1220 consensus_queue.cc:260] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.630506 1219 ts_tablet_manager.cc:1398] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:34.631093 1220 ts_tablet_manager.cc:1398] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:34.631299 1221 raft_consensus.cc:374] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:34.631403 1221 raft_consensus.cc:729] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b31a2332f794495092c03f45258a8ae2, State: Initialized, Role: FOLLOWER
I20240501 14:00:34.631652 1221 consensus_queue.cc:260] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.633116 1221 ts_tablet_manager.cc:1398] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:34.706354 1225 raft_consensus.cc:482] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:34.706547 1225 raft_consensus.cc:504] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.707278 1225 leader_election.cc:288] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b31a2332f794495092c03f45258a8ae2 (127.29.194.195:46007), dd4606e1b8cb456bb1a53dbaf04546fc (127.29.194.194:33699)
I20240501 14:00:34.715181 1173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18fb36808034d38a643426dbbde0748" candidate_uuid: "3ae7c0b58b514561914cee013441a2b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31a2332f794495092c03f45258a8ae2" is_pre_election: true
I20240501 14:00:34.715181 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18fb36808034d38a643426dbbde0748" candidate_uuid: "3ae7c0b58b514561914cee013441a2b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" is_pre_election: true
I20240501 14:00:34.715488 1173 raft_consensus.cc:2454] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae7c0b58b514561914cee013441a2b2 in term 0.
I20240501 14:00:34.715493 1099 raft_consensus.cc:2454] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ae7c0b58b514561914cee013441a2b2 in term 0.
I20240501 14:00:34.715927 988 leader_election.cc:302] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ae7c0b58b514561914cee013441a2b2, b31a2332f794495092c03f45258a8ae2, dd4606e1b8cb456bb1a53dbaf04546fc; no voters:
I20240501 14:00:34.716193 1225 raft_consensus.cc:2781] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:34.716284 1225 raft_consensus.cc:482] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:34.716367 1225 raft_consensus.cc:3037] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.718109 1225 raft_consensus.cc:504] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.718566 1225 leader_election.cc:288] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [CANDIDATE]: Term 1 election: Requested vote from peers b31a2332f794495092c03f45258a8ae2 (127.29.194.195:46007), dd4606e1b8cb456bb1a53dbaf04546fc (127.29.194.194:33699)
I20240501 14:00:34.718969 1099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18fb36808034d38a643426dbbde0748" candidate_uuid: "3ae7c0b58b514561914cee013441a2b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc"
I20240501 14:00:34.718974 1173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e18fb36808034d38a643426dbbde0748" candidate_uuid: "3ae7c0b58b514561914cee013441a2b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b31a2332f794495092c03f45258a8ae2"
I20240501 14:00:34.719130 1099 raft_consensus.cc:3037] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.719177 1173 raft_consensus.cc:3037] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:34.720916 1099 raft_consensus.cc:2454] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae7c0b58b514561914cee013441a2b2 in term 1.
I20240501 14:00:34.721046 1173 raft_consensus.cc:2454] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ae7c0b58b514561914cee013441a2b2 in term 1.
I20240501 14:00:34.721220 989 leader_election.cc:302] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ae7c0b58b514561914cee013441a2b2, dd4606e1b8cb456bb1a53dbaf04546fc; no voters:
I20240501 14:00:34.721485 1225 raft_consensus.cc:2781] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:34.721827 1225 raft_consensus.cc:686] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 1 LEADER]: Becoming Leader. State: Replica: 3ae7c0b58b514561914cee013441a2b2, State: Running, Role: LEADER
I20240501 14:00:34.722076 1225 consensus_queue.cc:237] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } }
I20240501 14:00:34.724340 921 catalog_manager.cc:5484] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ae7c0b58b514561914cee013441a2b2 (127.29.194.193). New cstate: current_term: 1 leader_uuid: "3ae7c0b58b514561914cee013441a2b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ae7c0b58b514561914cee013441a2b2" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 41923 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:35.232188 1232 consensus_queue.cc:1035] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b31a2332f794495092c03f45258a8ae2" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.237311 1238 consensus_queue.cc:1035] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4606e1b8cb456bb1a53dbaf04546fc" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 33699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:35.342988 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:35.343452 975 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:35.343961 975 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:35.344102 975 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:36.345973 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:37.344579 975 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:37.345005 975 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:37.345099 975 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:37.347954 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:38.350291 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:39.345590 975 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:39.346031 975 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:39.346128 975 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:39.352560 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:40.354910 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:41.346624 975 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:41.347069 975 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:41.347174 975 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:41.357285 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:42.359620 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:43.347656 975 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:43.348120 975 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:43.348219 975 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:43.361935 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:44.364327 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:45.348655 975 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:45.349087 975 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:45.349180 975 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:45.366714 970 placement_policy_util.cc:407] tablet e18fb36808034d38a643426dbbde0748: detected majority of replicas (2 of 3) at location L0
I20240501 14:00:45.785914 30475 tablet_server.cc:178] TabletServer@127.29.194.193:0 shutting down...
I20240501 14:00:45.800384 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:45.800630 30475 tablet_replica.cc:330] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2: stopping tablet replica
I20240501 14:00:45.800845 30475 raft_consensus.cc:2229] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.801141 30475 raft_consensus.cc:2258] T e18fb36808034d38a643426dbbde0748 P 3ae7c0b58b514561914cee013441a2b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.814570 30475 tablet_server.cc:194] TabletServer@127.29.194.193:0 shutdown complete.
I20240501 14:00:45.817211 30475 tablet_server.cc:178] TabletServer@127.29.194.194:0 shutting down...
I20240501 14:00:45.824642 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:45.824887 30475 tablet_replica.cc:330] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc: stopping tablet replica
I20240501 14:00:45.825050 30475 raft_consensus.cc:2229] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.825201 30475 raft_consensus.cc:2258] T e18fb36808034d38a643426dbbde0748 P dd4606e1b8cb456bb1a53dbaf04546fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.838173 30475 tablet_server.cc:194] TabletServer@127.29.194.194:0 shutdown complete.
I20240501 14:00:45.840920 30475 tablet_server.cc:178] TabletServer@127.29.194.195:0 shutting down...
I20240501 14:00:45.848069 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:00:45.848364 30475 tablet_replica.cc:330] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2: stopping tablet replica
I20240501 14:00:45.848508 30475 raft_consensus.cc:2229] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:45.848660 30475 raft_consensus.cc:2258] T e18fb36808034d38a643426dbbde0748 P b31a2332f794495092c03f45258a8ae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.861672 30475 tablet_server.cc:194] TabletServer@127.29.194.195:0 shutdown complete.
I20240501 14:00:45.864394 30475 master.cc:537] Master@127.29.194.254:35931 shutting down...
I20240501 14:00:45.871073 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:00:45.871274 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:45.871376 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 26695e81fba9443e99f5758e24996b15: stopping tablet replica
I20240501 14:00:45.884229 30475 master.cc:552] Master@127.29.194.254:35931 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12609 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240501 14:00:45.894409 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:37029
I20240501 14:00:45.894840 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:45.897712 1244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:45.898401 1243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:45.899016 1246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:45.899646 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:45.899878 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:45.899925 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:45.899958 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572045899954 us; error 0 us; skew 500 ppm
I20240501 14:00:45.900121 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:45.906469 30475 webserver.cc:460] Webserver started at http://127.29.194.254:38139/ using document root <none> and password file <none>
I20240501 14:00:45.906685 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:45.906749 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:45.906870 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:45.907503 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "1e2f523513904e35ad8e2accc3750429"
format_stamp: "Formatted at 2024-05-01 14:00:45 on dist-test-slave-kwk7"
I20240501 14:00:45.909955 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:45.911412 1251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:45.911727 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:45.911815 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "1e2f523513904e35ad8e2accc3750429"
format_stamp: "Formatted at 2024-05-01 14:00:45 on dist-test-slave-kwk7"
I20240501 14:00:45.911917 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:45.922736 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:45.923316 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:45.934784 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:37029
I20240501 14:00:45.934805 1302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:37029 every 8 connection(s)
I20240501 14:00:45.935989 1303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:45.939819 1303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429: Bootstrap starting.
I20240501 14:00:45.941147 1303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:45.942699 1303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429: No bootstrap required, opened a new log
I20240501 14:00:45.943245 1303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f523513904e35ad8e2accc3750429" member_type: VOTER }
I20240501 14:00:45.943365 1303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:45.943414 1303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2f523513904e35ad8e2accc3750429, State: Initialized, Role: FOLLOWER
I20240501 14:00:45.943598 1303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f523513904e35ad8e2accc3750429" member_type: VOTER }
I20240501 14:00:45.943702 1303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:00:45.943756 1303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:00:45.943820 1303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:45.945539 1303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f523513904e35ad8e2accc3750429" member_type: VOTER }
I20240501 14:00:45.945713 1303 leader_election.cc:302] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e2f523513904e35ad8e2accc3750429; no voters:
I20240501 14:00:45.946038 1303 leader_election.cc:288] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:00:45.946143 1306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:45.946548 1306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2f523513904e35ad8e2accc3750429, State: Running, Role: LEADER
I20240501 14:00:45.946750 1306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f523513904e35ad8e2accc3750429" member_type: VOTER }
I20240501 14:00:45.946810 1303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:00:45.947381 1307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e2f523513904e35ad8e2accc3750429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f523513904e35ad8e2accc3750429" member_type: VOTER } }
I20240501 14:00:45.947602 1307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:45.947434 1308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e2f523513904e35ad8e2accc3750429. Latest consensus state: current_term: 1 leader_uuid: "1e2f523513904e35ad8e2accc3750429" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2f523513904e35ad8e2accc3750429" member_type: VOTER } }
I20240501 14:00:45.947911 1308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [sys.catalog]: This master's current role is: LEADER
I20240501 14:00:45.951248 1311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:00:45.951550 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:00:45.952584 1311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:00:45.955058 1311 catalog_manager.cc:1347] Generated new cluster ID: 64f05c7f868c4d3e9c38044b2e3f6183
I20240501 14:00:45.955121 1311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:00:45.959837 1311 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:00:45.960533 1311 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:00:45.966954 1311 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429: Generated new TSK 0
I20240501 14:00:45.967149 1311 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:00:45.984253 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:45.987577 1324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:45.988344 1325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:45.988996 1327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:45.989612 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:45.989893 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:45.989949 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:45.989995 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572045989990 us; error 0 us; skew 500 ppm
I20240501 14:00:45.990139 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:45.991335 30475 webserver.cc:460] Webserver started at http://127.29.194.193:33127/ using document root <none> and password file <none>
I20240501 14:00:45.991581 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:45.991648 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:45.991766 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:45.992383 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "c07691f747984bc7a2df956f4f212d8e"
format_stamp: "Formatted at 2024-05-01 14:00:45 on dist-test-slave-kwk7"
I20240501 14:00:45.994987 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:45.996536 1332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:45.996960 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:45.997056 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "c07691f747984bc7a2df956f4f212d8e"
format_stamp: "Formatted at 2024-05-01 14:00:45 on dist-test-slave-kwk7"
I20240501 14:00:45.997144 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:46.029701 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:46.030265 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:46.030784 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:46.031615 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:46.031668 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:46.031740 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:46.031781 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:46.043905 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:46821
I20240501 14:00:46.043923 1394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:46821 every 8 connection(s)
I20240501 14:00:46.045629 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:46.049129 1399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:46.049825 1400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:46.050441 1402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:46.051074 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:46.051363 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:46.051442 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:46.051472 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572046051469 us; error 0 us; skew 500 ppm
I20240501 14:00:46.051614 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:46.052438 1395 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37029
I20240501 14:00:46.052582 1395 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:46.053010 1395 heartbeater.cc:511] Master 127.29.194.254:37029 requested a full tablet report, sending...
I20240501 14:00:46.053010 30475 webserver.cc:460] Webserver started at http://127.29.194.194:42499/ using document root <none> and password file <none>
I20240501 14:00:46.053231 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:46.053298 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:46.053404 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:46.053884 1268 ts_manager.cc:194] Registered new tserver with Master: c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:00:46.054141 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "31d5b9d4a2d2406a972f48086b62ceed"
format_stamp: "Formatted at 2024-05-01 14:00:46 on dist-test-slave-kwk7"
I20240501 14:00:46.055392 1268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53602
I20240501 14:00:46.056911 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:46.058207 1407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:46.058480 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:46.058563 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "31d5b9d4a2d2406a972f48086b62ceed"
format_stamp: "Formatted at 2024-05-01 14:00:46 on dist-test-slave-kwk7"
I20240501 14:00:46.058663 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:46.067638 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:46.068140 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:46.068609 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:46.069296 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:46.069347 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:46.069412 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:46.069445 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:46.079921 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:38379
I20240501 14:00:46.079948 1469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:38379 every 8 connection(s)
I20240501 14:00:46.081394 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:46.084601 1473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:46.085244 1474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:46.085827 30475 server_base.cc:1034] running on GCE node
W20240501 14:00:46.086359 1476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:46.086730 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:46.086805 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:46.086849 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572046086846 us; error 0 us; skew 500 ppm
I20240501 14:00:46.086995 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:46.087118 1470 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37029
I20240501 14:00:46.087246 1470 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:46.087586 1470 heartbeater.cc:511] Master 127.29.194.254:37029 requested a full tablet report, sending...
I20240501 14:00:46.088297 30475 webserver.cc:460] Webserver started at http://127.29.194.195:36887/ using document root <none> and password file <none>
I20240501 14:00:46.088459 1268 ts_manager.cc:194] Registered new tserver with Master: 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:46.088510 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:46.088578 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:46.088707 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:46.089409 1268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53614
I20240501 14:00:46.089409 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "1e896a0495ca458abad35d883aef4aef"
format_stamp: "Formatted at 2024-05-01 14:00:46 on dist-test-slave-kwk7"
I20240501 14:00:46.091809 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:46.093235 1481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:46.093515 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:46.093617 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "1e896a0495ca458abad35d883aef4aef"
format_stamp: "Formatted at 2024-05-01 14:00:46 on dist-test-slave-kwk7"
I20240501 14:00:46.093720 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:46.131098 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:46.131575 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:46.132074 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:46.132882 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:46.132937 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:46.133019 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:46.133057 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:46.142747 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:38049
I20240501 14:00:46.142777 1543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:38049 every 8 connection(s)
I20240501 14:00:46.148263 1544 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37029
I20240501 14:00:46.148378 1544 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:46.148667 1544 heartbeater.cc:511] Master 127.29.194.254:37029 requested a full tablet report, sending...
I20240501 14:00:46.149425 1268 ts_manager.cc:194] Registered new tserver with Master: 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:46.150104 1268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53620
I20240501 14:00:46.150274 30475 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006719093s
I20240501 14:00:47.056543 1395 heartbeater.cc:503] Master 127.29.194.254:37029 was elected leader, sending a full tablet report...
I20240501 14:00:47.090536 1470 heartbeater.cc:503] Master 127.29.194.254:37029 was elected leader, sending a full tablet report...
I20240501 14:00:47.151355 1544 heartbeater.cc:503] Master 127.29.194.254:37029 was elected leader, sending a full tablet report...
I20240501 14:00:47.177456 30475 test_util.cc:268] Using random seed: -372222245
I20240501 14:00:47.189982 1268 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53636:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:00:47.190762 1268 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:00:47.224820 1360 tablet_service.cc:1467] Processing CreateTablet for tablet d3b31989cdae4e9ab72ad37c9e61f903 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 14:00:47.225311 1360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b31989cdae4e9ab72ad37c9e61f903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.225503 1359 tablet_service.cc:1467] Processing CreateTablet for tablet 78e839ad73564640b24fe9efc9f41f08 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 14:00:47.225821 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e839ad73564640b24fe9efc9f41f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.225965 1352 tablet_service.cc:1467] Processing CreateTablet for tablet 407fed285f274f3c95342a21421b2c66 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 14:00:47.226351 1352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407fed285f274f3c95342a21421b2c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.228987 1358 tablet_service.cc:1467] Processing CreateTablet for tablet 916a3f31691044f783d655695bf0a369 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 14:00:47.229357 1358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916a3f31691044f783d655695bf0a369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.232488 1352 tablet_service.cc:1467] Processing CreateTablet for tablet 2bdda4e923dc4caaba534f1e8b5693e2 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 14:00:47.232604 1351 tablet_service.cc:1467] Processing CreateTablet for tablet 348221e7dbd04efcba6185ba29a33a60 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 14:00:47.232890 1352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bdda4e923dc4caaba534f1e8b5693e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.233129 1351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348221e7dbd04efcba6185ba29a33a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.235874 1509 tablet_service.cc:1467] Processing CreateTablet for tablet d3b31989cdae4e9ab72ad37c9e61f903 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 14:00:47.236166 1509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b31989cdae4e9ab72ad37c9e61f903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.236948 1357 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 14:00:47.237239 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee8e1d3ee3b429ebd15a32afe393e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.237603 1426 tablet_service.cc:1467] Processing CreateTablet for tablet 2bdda4e923dc4caaba534f1e8b5693e2 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 14:00:47.237943 1426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bdda4e923dc4caaba534f1e8b5693e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.238641 1352 tablet_service.cc:1467] Processing CreateTablet for tablet ae2d33db85414876998a3ed550e44e41 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 14:00:47.238924 1352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2d33db85414876998a3ed550e44e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.240943 1427 tablet_service.cc:1467] Processing CreateTablet for tablet 407fed285f274f3c95342a21421b2c66 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 14:00:47.240998 1508 tablet_service.cc:1467] Processing CreateTablet for tablet 78e839ad73564640b24fe9efc9f41f08 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 14:00:47.241288 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407fed285f274f3c95342a21421b2c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.241389 1508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e839ad73564640b24fe9efc9f41f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.242841 1564 tablet_bootstrap.cc:492] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.244251 1564 tablet_bootstrap.cc:654] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.244990 1507 tablet_service.cc:1467] Processing CreateTablet for tablet 916a3f31691044f783d655695bf0a369 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 14:00:47.245290 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 355ae59edd1c4620904ba23275021582 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 14:00:47.245548 1507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916a3f31691044f783d655695bf0a369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.245698 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355ae59edd1c4620904ba23275021582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.248656 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 46ae855dc1a44159b5f3cca0657dc752 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 14:00:47.249249 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ae855dc1a44159b5f3cca0657dc752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.249104 1504 tablet_service.cc:1467] Processing CreateTablet for tablet 355ae59edd1c4620904ba23275021582 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 14:00:47.249624 1504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355ae59edd1c4620904ba23275021582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.253115 1503 tablet_service.cc:1467] Processing CreateTablet for tablet 46ae855dc1a44159b5f3cca0657dc752 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 14:00:47.253489 1566 tablet_bootstrap.cc:492] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.253441 1435 tablet_service.cc:1467] Processing CreateTablet for tablet d3b31989cdae4e9ab72ad37c9e61f903 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 14:00:47.253551 1503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ae855dc1a44159b5f3cca0657dc752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.253829 1435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3b31989cdae4e9ab72ad37c9e61f903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.254055 1427 tablet_service.cc:1467] Processing CreateTablet for tablet 348221e7dbd04efcba6185ba29a33a60 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 14:00:47.254448 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348221e7dbd04efcba6185ba29a33a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.256338 1566 tablet_bootstrap.cc:654] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.257058 1434 tablet_service.cc:1467] Processing CreateTablet for tablet 78e839ad73564640b24fe9efc9f41f08 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 14:00:47.257171 1502 tablet_service.cc:1467] Processing CreateTablet for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 14:00:47.257963 1502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c36a1a8c09d415d8c1d1860c0d8b622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.258316 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 916a3f31691044f783d655695bf0a369 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 14:00:47.258932 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 14:00:47.259549 1431 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca80506974245cb887559f7f884fb98 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 14:00:47.260176 1430 tablet_service.cc:1467] Processing CreateTablet for tablet 355ae59edd1c4620904ba23275021582 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 14:00:47.260789 1429 tablet_service.cc:1467] Processing CreateTablet for tablet 46ae855dc1a44159b5f3cca0657dc752 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 14:00:47.260965 1427 tablet_service.cc:1467] Processing CreateTablet for tablet ae2d33db85414876998a3ed550e44e41 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 14:00:47.261382 1501 tablet_service.cc:1467] Processing CreateTablet for tablet 407fed285f274f3c95342a21421b2c66 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 14:00:47.245002 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca80506974245cb887559f7f884fb98 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 14:00:47.257967 1434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e839ad73564640b24fe9efc9f41f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.261855 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca80506974245cb887559f7f884fb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.261931 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407fed285f274f3c95342a21421b2c66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.264811 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee8e1d3ee3b429ebd15a32afe393e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.265017 1500 tablet_service.cc:1467] Processing CreateTablet for tablet 2bdda4e923dc4caaba534f1e8b5693e2 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 14:00:47.265254 1566 tablet_bootstrap.cc:492] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.265403 1500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bdda4e923dc4caaba534f1e8b5693e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.265388 1566 ts_tablet_manager.cc:1367] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240501 14:00:47.266654 1566 raft_consensus.cc:348] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.266834 1566 raft_consensus.cc:374] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.266878 1566 raft_consensus.cc:729] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.268218 1429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ae855dc1a44159b5f3cca0657dc752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.268103 1566 consensus_queue.cc:260] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.268783 1431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca80506974245cb887559f7f884fb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.269631 1505 tablet_service.cc:1467] Processing CreateTablet for tablet 7ca80506974245cb887559f7f884fb98 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 14:00:47.270171 1505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca80506974245cb887559f7f884fb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.270977 1430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355ae59edd1c4620904ba23275021582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.245002 1428 tablet_service.cc:1467] Processing CreateTablet for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 14:00:47.271482 1428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c36a1a8c09d415d8c1d1860c0d8b622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.253124 1353 tablet_service.cc:1467] Processing CreateTablet for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 14:00:47.273415 1353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c36a1a8c09d415d8c1d1860c0d8b622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.274791 1566 ts_tablet_manager.cc:1398] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.009s user 0.001s sys 0.003s
I20240501 14:00:47.276039 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916a3f31691044f783d655695bf0a369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.276046 1566 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.248678 1506 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 14:00:47.277228 1506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee8e1d3ee3b429ebd15a32afe393e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.279302 1566 tablet_bootstrap.cc:654] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.281643 1566 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.281790 1566 ts_tablet_manager.cc:1367] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:00:47.282442 1566 raft_consensus.cc:348] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.282629 1566 raft_consensus.cc:374] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.284068 1566 raft_consensus.cc:729] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.284475 1566 consensus_queue.cc:260] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.285847 1566 ts_tablet_manager.cc:1398] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:47.286386 1566 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.288640 1566 tablet_bootstrap.cc:654] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.293020 1566 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.293212 1566 ts_tablet_manager.cc:1367] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:00:47.293948 1566 raft_consensus.cc:348] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.294565 1566 raft_consensus.cc:374] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.294762 1566 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.295145 1566 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.298010 1566 ts_tablet_manager.cc:1398] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:00:47.299801 1500 tablet_service.cc:1467] Processing CreateTablet for tablet 348221e7dbd04efcba6185ba29a33a60 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 14:00:47.300158 1566 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.315315 1500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348221e7dbd04efcba6185ba29a33a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.300488 1505 tablet_service.cc:1467] Processing CreateTablet for tablet ae2d33db85414876998a3ed550e44e41 (DEFAULT_TABLE table=test-workload [id=abdbc8974103469498ec20bb03452b7f]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 14:00:47.304386 1568 raft_consensus.cc:482] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.315742 1568 raft_consensus.cc:504] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.316763 1568 leader_election.cc:288] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.319206 1505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2d33db85414876998a3ed550e44e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.307274 1427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2d33db85414876998a3ed550e44e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:47.323495 1566 tablet_bootstrap.cc:654] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.326122 1571 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.329267 1571 tablet_bootstrap.cc:654] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.330997 1564 tablet_bootstrap.cc:492] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.331501 1564 ts_tablet_manager.cc:1367] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.089s user 0.012s sys 0.040s
I20240501 14:00:47.331734 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
W20240501 14:00:47.332525 1408 leader_election.cc:341] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.332924 1566 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.332924 1571 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.333154 1566 ts_tablet_manager.cc:1367] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.005s
I20240501 14:00:47.333222 1571 ts_tablet_manager.cc:1367] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240501 14:00:47.333452 1564 raft_consensus.cc:348] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.333694 1564 raft_consensus.cc:374] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.333796 1564 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.333835 1571 raft_consensus.cc:348] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.333884 1566 raft_consensus.cc:348] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.334064 1571 raft_consensus.cc:374] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.334156 1566 raft_consensus.cc:374] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.334228 1571 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.334033 1564 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.334295 1566 raft_consensus.cc:729] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.334478 1571 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.334630 1566 consensus_queue.cc:260] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.338150 1566 ts_tablet_manager.cc:1398] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 14:00:47.338570 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef" is_pre_election: true
W20240501 14:00:47.339354 1409 leader_election.cc:341] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.339546 1409 leader_election.cc:302] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed; no voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:47.340552 1568 raft_consensus.cc:2726] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:00:47.341078 1566 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.341394 1571 ts_tablet_manager.cc:1398] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20240501 14:00:47.341720 1571 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.342445 1566 tablet_bootstrap.cc:654] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.343019 1571 tablet_bootstrap.cc:654] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.346961 1564 ts_tablet_manager.cc:1398] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20240501 14:00:47.347522 1566 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.347708 1566 ts_tablet_manager.cc:1367] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240501 14:00:47.348505 1566 raft_consensus.cc:348] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.348780 1566 raft_consensus.cc:374] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.348855 1566 raft_consensus.cc:729] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.349169 1566 consensus_queue.cc:260] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.349793 1566 ts_tablet_manager.cc:1398] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:47.350080 1566 tablet_bootstrap.cc:492] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.351331 1566 tablet_bootstrap.cc:654] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.351935 1571 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.351967 1564 tablet_bootstrap.cc:492] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.352087 1571 ts_tablet_manager.cc:1367] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 14:00:47.353282 1571 raft_consensus.cc:348] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.353541 1564 tablet_bootstrap.cc:654] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.353514 1571 raft_consensus.cc:374] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.353648 1571 raft_consensus.cc:729] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.353864 1571 consensus_queue.cc:260] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.354348 1571 ts_tablet_manager.cc:1398] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:47.354557 1571 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.355252 1566 tablet_bootstrap.cc:492] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.355388 1566 ts_tablet_manager.cc:1367] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:00:47.355751 1571 tablet_bootstrap.cc:654] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.356047 1566 raft_consensus.cc:348] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.356242 1566 raft_consensus.cc:374] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.356350 1566 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.356578 1566 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.357184 1566 ts_tablet_manager.cc:1398] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:47.357471 1566 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.357527 1571 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.357700 1571 ts_tablet_manager.cc:1367] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:47.358211 1571 raft_consensus.cc:348] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.358340 1571 raft_consensus.cc:374] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.358422 1571 raft_consensus.cc:729] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.358598 1571 consensus_queue.cc:260] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.358739 1566 tablet_bootstrap.cc:654] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.358952 1571 ts_tablet_manager.cc:1398] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:47.359655 1571 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.360630 1571 tablet_bootstrap.cc:654] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.365798 1566 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.365959 1566 ts_tablet_manager.cc:1367] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.001s
I20240501 14:00:47.366251 1564 tablet_bootstrap.cc:492] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.366371 1564 ts_tablet_manager.cc:1367] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20240501 14:00:47.366593 1571 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.366716 1571 ts_tablet_manager.cc:1367] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240501 14:00:47.366621 1566 raft_consensus.cc:348] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.366783 1566 raft_consensus.cc:374] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.366845 1566 raft_consensus.cc:729] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.366911 1564 raft_consensus.cc:348] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.367086 1564 raft_consensus.cc:374] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.367022 1566 consensus_queue.cc:260] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.367147 1564 raft_consensus.cc:729] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.367285 1571 raft_consensus.cc:348] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.367429 1571 raft_consensus.cc:374] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.367354 1564 consensus_queue.cc:260] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.367475 1571 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.367483 1566 ts_tablet_manager.cc:1398] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:47.367731 1566 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.367676 1571 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.368086 1571 ts_tablet_manager.cc:1398] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:00:47.368316 1571 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.369159 1566 tablet_bootstrap.cc:654] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.369827 1571 tablet_bootstrap.cc:654] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.373553 1576 raft_consensus.cc:482] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.373773 1576 raft_consensus.cc:504] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.377341 1564 ts_tablet_manager.cc:1398] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240501 14:00:47.377660 1564 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.380604 1564 tablet_bootstrap.cc:654] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.385577 1576 leader_election.cc:288] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.411923 1564 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.412108 1564 ts_tablet_manager.cc:1367] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.013s
I20240501 14:00:47.413101 1564 raft_consensus.cc:348] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.413321 1564 raft_consensus.cc:374] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.413388 1564 raft_consensus.cc:729] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.413666 1564 consensus_queue.cc:260] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.413753 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.414121 1445 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 0.
I20240501 14:00:47.414253 1564 ts_tablet_manager.cc:1398] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:47.414710 1336 leader_election.cc:302] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.415019 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef" is_pre_election: true
W20240501 14:00:47.415575 1334 leader_election.cc:341] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.415942 1585 raft_consensus.cc:2781] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.416091 1585 raft_consensus.cc:482] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.416222 1585 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.418040 1564 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.419538 1564 tablet_bootstrap.cc:654] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.419679 1585 raft_consensus.cc:504] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.420861 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.421118 1445 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.425186 1445 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 1.
I20240501 14:00:47.425683 1336 leader_election.cc:302] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.425983 1576 raft_consensus.cc:2781] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.432793 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef"
W20240501 14:00:47.433455 1334 leader_election.cc:341] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.435662 1566 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.435797 1566 ts_tablet_manager.cc:1367] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.068s user 0.039s sys 0.023s
I20240501 14:00:47.436066 1585 leader_election.cc:288] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.436445 1571 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.436635 1571 ts_tablet_manager.cc:1367] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.068s user 0.024s sys 0.043s
I20240501 14:00:47.436532 1566 raft_consensus.cc:348] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.436821 1566 raft_consensus.cc:374] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.436920 1566 raft_consensus.cc:729] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.437194 1566 consensus_queue.cc:260] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.437358 1571 raft_consensus.cc:348] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.437528 1571 raft_consensus.cc:374] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.437579 1571 raft_consensus.cc:729] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.437763 1571 consensus_queue.cc:260] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.438241 1571 ts_tablet_manager.cc:1398] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:00:47.438583 1571 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.440228 1566 ts_tablet_manager.cc:1398] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:00:47.440321 1571 tablet_bootstrap.cc:654] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.442471 1566 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.444859 1566 tablet_bootstrap.cc:654] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.445098 1576 raft_consensus.cc:686] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:47.445528 1576 consensus_queue.cc:237] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.448737 1571 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.448909 1571 ts_tablet_manager.cc:1367] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240501 14:00:47.449852 1571 raft_consensus.cc:348] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.450157 1571 raft_consensus.cc:374] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.450253 1571 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.450552 1571 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.450853 1568 raft_consensus.cc:482] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.451006 1568 raft_consensus.cc:504] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.451682 1568 leader_election.cc:288] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.451583 1267 catalog_manager.cc:5484] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 1 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.452293 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ae855dc1a44159b5f3cca0657dc752" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef" is_pre_election: true
I20240501 14:00:47.452576 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ae855dc1a44159b5f3cca0657dc752" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
W20240501 14:00:47.453027 1408 leader_election.cc:341] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.453464 1519 raft_consensus.cc:2454] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 0.
I20240501 14:00:47.455158 1571 ts_tablet_manager.cc:1398] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:47.455164 1409 leader_election.cc:302] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters: c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:47.455698 1568 raft_consensus.cc:2781] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.456240 1568 raft_consensus.cc:482] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.456336 1568 raft_consensus.cc:3037] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.457104 1566 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.457247 1566 ts_tablet_manager.cc:1367] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20240501 14:00:47.457417 1564 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.457598 1564 ts_tablet_manager.cc:1367] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.040s user 0.014s sys 0.018s
I20240501 14:00:47.458025 1566 raft_consensus.cc:348] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.458202 1564 raft_consensus.cc:348] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.458691 1566 raft_consensus.cc:374] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.458695 1564 raft_consensus.cc:374] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.458794 1566 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.458896 1564 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.459028 1566 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.459122 1564 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.459672 1564 ts_tablet_manager.cc:1398] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:47.459842 1568 raft_consensus.cc:504] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.460044 1564 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.455448 1571 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.461031 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ae855dc1a44159b5f3cca0657dc752" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
W20240501 14:00:47.461544 1408 leader_election.cc:341] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.462821 1564 tablet_bootstrap.cc:654] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.463093 1571 tablet_bootstrap.cc:654] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.464617 1568 leader_election.cc:288] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.465010 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ae855dc1a44159b5f3cca0657dc752" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:47.465276 1519 raft_consensus.cc:3037] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.465902 1566 ts_tablet_manager.cc:1398] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20240501 14:00:47.466188 1566 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.467904 1566 tablet_bootstrap.cc:654] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.469820 1519 raft_consensus.cc:2454] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 1.
I20240501 14:00:47.470373 1409 leader_election.cc:302] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters: c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:47.470911 1568 raft_consensus.cc:2781] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.471009 1571 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.471010 1564 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.471302 1564 ts_tablet_manager.cc:1367] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240501 14:00:47.471312 1571 ts_tablet_manager.cc:1367] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240501 14:00:47.471113 1568 raft_consensus.cc:686] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:47.472011 1568 consensus_queue.cc:237] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.472103 1564 raft_consensus.cc:348] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.472317 1564 raft_consensus.cc:374] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.472401 1564 raft_consensus.cc:729] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.472208 1571 raft_consensus.cc:348] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.472590 1571 raft_consensus.cc:374] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.472759 1571 raft_consensus.cc:729] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.472755 1564 consensus_queue.cc:260] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.473104 1571 consensus_queue.cc:260] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.473662 1571 ts_tablet_manager.cc:1398] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:47.474013 1571 tablet_bootstrap.cc:492] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.474040 1564 ts_tablet_manager.cc:1398] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:00:47.474378 1564 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.476787 1571 tablet_bootstrap.cc:654] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.477391 1564 tablet_bootstrap.cc:654] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.479143 1566 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.479316 1566 ts_tablet_manager.cc:1367] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240501 14:00:47.480019 1566 raft_consensus.cc:348] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.480307 1566 raft_consensus.cc:374] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.480413 1566 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.480742 1566 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.481446 1566 ts_tablet_manager.cc:1398] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:00:47.481801 1566 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.483922 1566 tablet_bootstrap.cc:654] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.486073 1571 tablet_bootstrap.cc:492] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.486249 1571 ts_tablet_manager.cc:1367] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20240501 14:00:47.486968 1571 raft_consensus.cc:348] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.487175 1571 raft_consensus.cc:374] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.487262 1571 raft_consensus.cc:729] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.487061 1268 catalog_manager.cc:5484] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 0 to 1, leader changed from <none> to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 1 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.487565 1571 consensus_queue.cc:260] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.488040 1571 ts_tablet_manager.cc:1398] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:47.488314 1571 tablet_bootstrap.cc:492] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.489828 1571 tablet_bootstrap.cc:654] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.492672 1571 tablet_bootstrap.cc:492] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.493005 1571 ts_tablet_manager.cc:1367] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240501 14:00:47.493521 1566 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.493681 1566 ts_tablet_manager.cc:1367] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.001s
I20240501 14:00:47.494066 1577 raft_consensus.cc:482] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.494220 1577 raft_consensus.cc:504] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.494421 1566 raft_consensus.cc:348] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.494621 1566 raft_consensus.cc:374] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.494719 1566 raft_consensus.cc:729] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.494339 1571 raft_consensus.cc:348] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.494940 1571 raft_consensus.cc:374] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.494937 1566 consensus_queue.cc:260] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.495256 1571 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.495568 1566 ts_tablet_manager.cc:1398] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:47.495821 1566 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: Bootstrap starting.
I20240501 14:00:47.495702 1571 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.497535 1566 tablet_bootstrap.cc:654] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.495321 1577 leader_election.cc:288] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.500344 1564 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.500536 1564 ts_tablet_manager.cc:1367] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.012s
I20240501 14:00:47.501441 1564 raft_consensus.cc:348] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.501726 1564 raft_consensus.cc:374] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.501819 1564 raft_consensus.cc:729] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.502139 1564 consensus_queue.cc:260] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.503901 1564 ts_tablet_manager.cc:1398] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:00:47.504254 1564 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.506395 1564 tablet_bootstrap.cc:654] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.545257 1576 raft_consensus.cc:482] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.545497 1576 raft_consensus.cc:504] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.546257 1576 leader_election.cc:288] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.547632 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.547931 1445 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 0.
I20240501 14:00:47.548439 1336 leader_election.cc:302] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.549166 1576 raft_consensus.cc:2781] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.549296 1576 raft_consensus.cc:482] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.549417 1576 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.554261 1576 raft_consensus.cc:504] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.555433 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.555670 1445 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.559649 1445 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 1.
I20240501 14:00:47.560111 1336 leader_election.cc:302] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.560496 1592 raft_consensus.cc:2781] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.560595 1577 raft_consensus.cc:482] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.560739 1592 raft_consensus.cc:686] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:47.560832 1577 raft_consensus.cc:504] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.561231 1592 consensus_queue.cc:237] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.562683 1577 leader_election.cc:288] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.564229 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef" is_pre_election: true
I20240501 14:00:47.564471 1519 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 0.
I20240501 14:00:47.566602 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:47.566823 1519 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.570751 1519 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 1.
I20240501 14:00:47.571684 1576 leader_election.cc:288] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 1 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.576769 1566 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: No bootstrap required, opened a new log
I20240501 14:00:47.576933 1566 ts_tablet_manager.cc:1367] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent bootstrapping tablet: real 0.081s user 0.044s sys 0.036s
I20240501 14:00:47.577769 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca80506974245cb887559f7f884fb98" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.577723 1566 raft_consensus.cc:348] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.577960 1566 raft_consensus.cc:374] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.578034 1566 raft_consensus.cc:729] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.578279 1566 consensus_queue.cc:260] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.578477 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.578675 1370 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
W20240501 14:00:47.579075 1482 leader_election.cc:341] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.579283 1482 leader_election.cc:302] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.579540 1577 raft_consensus.cc:2781] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.579643 1577 raft_consensus.cc:482] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.579739 1577 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.579968 1566 ts_tablet_manager.cc:1398] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:00:47.582700 1577 raft_consensus.cc:504] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.588922 1606 raft_consensus.cc:482] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.589229 1571 ts_tablet_manager.cc:1398] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.096s user 0.030s sys 0.059s
I20240501 14:00:47.589094 1606 raft_consensus.cc:504] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.589402 1564 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.589570 1564 ts_tablet_manager.cc:1367] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.085s user 0.023s sys 0.059s
I20240501 14:00:47.589767 1571 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.589810 1268 catalog_manager.cc:5484] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 0 to 1, leader changed from <none> to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 1 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.590250 1606 leader_election.cc:288] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.590368 1564 raft_consensus.cc:348] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.590564 1564 raft_consensus.cc:374] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.590651 1564 raft_consensus.cc:729] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.590927 1564 consensus_queue.cc:260] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.591537 1564 ts_tablet_manager.cc:1398] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:00:47.591827 1564 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.592222 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.593401 1571 tablet_bootstrap.cc:654] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.593436 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.593691 1370 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.594130 1564 tablet_bootstrap.cc:654] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.596462 1370 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.596808 1564 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.597102 1564 ts_tablet_manager.cc:1367] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
W20240501 14:00:47.597514 1482 leader_election.cc:341] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:00:47.597749 1482 leader_election.cc:302] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.598049 1606 raft_consensus.cc:2781] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.598016 1564 raft_consensus.cc:348] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.598289 1564 raft_consensus.cc:374] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.598377 1564 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.598680 1564 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.598768 1577 leader_election.cc:288] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.599328 1564 ts_tablet_manager.cc:1398] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:47.600323 1606 raft_consensus.cc:686] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:47.600672 1564 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.601249 1606 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.601871 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.602411 1444 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.602506 1564 tablet_bootstrap.cc:654] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.602738 1445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca80506974245cb887559f7f884fb98" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.602679 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.603343 1445 raft_consensus.cc:2454] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.602823 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.603477 1443 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.603562 1442 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.604823 1564 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.605002 1564 ts_tablet_manager.cc:1367] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 14:00:47.605533 1485 leader_election.cc:302] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters: c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:47.605738 1564 raft_consensus.cc:348] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.605917 1564 raft_consensus.cc:374] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.605966 1485 leader_election.cc:302] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters: c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:47.606029 1564 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.606357 1577 raft_consensus.cc:2781] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.606231 1564 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.606439 1577 raft_consensus.cc:482] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.606539 1577 raft_consensus.cc:3037] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.607872 1442 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.608990 1571 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.609159 1571 ts_tablet_manager.cc:1367] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20240501 14:00:47.609175 1606 raft_consensus.cc:2781] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.609288 1606 raft_consensus.cc:482] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.609419 1606 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.609359 1577 raft_consensus.cc:504] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.609920 1571 raft_consensus.cc:348] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.610141 1571 raft_consensus.cc:374] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.610242 1571 raft_consensus.cc:729] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.610404 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca80506974245cb887559f7f884fb98" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.610476 1571 consensus_queue.cc:260] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.610611 1370 raft_consensus.cc:3037] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.610957 1571 ts_tablet_manager.cc:1398] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:00:47.611056 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ca80506974245cb887559f7f884fb98" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.611279 1571 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.611279 1442 raft_consensus.cc:3037] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.612681 1606 raft_consensus.cc:504] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.613735 1571 tablet_bootstrap.cc:654] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.614013 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.614193 1370 raft_consensus.cc:2454] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.614255 1369 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.608984 1564 ts_tablet_manager.cc:1398] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:00:47.615475 1564 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.615540 1442 raft_consensus.cc:2454] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.615895 1485 leader_election.cc:302] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:47.616153 1613 raft_consensus.cc:2781] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.616556 1369 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.617161 1482 leader_election.cc:302] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.617399 1564 tablet_bootstrap.cc:654] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.617408 1617 raft_consensus.cc:2781] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.617604 1606 leader_election.cc:288] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.618023 1617 raft_consensus.cc:686] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:47.618606 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.618819 1443 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.619463 1571 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.619616 1571 ts_tablet_manager.cc:1367] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:00:47.619866 1268 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.620443 1571 raft_consensus.cc:348] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.620649 1571 raft_consensus.cc:374] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.620781 1571 raft_consensus.cc:729] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.621027 1571 consensus_queue.cc:260] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.621299 1613 raft_consensus.cc:686] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:47.621578 1571 ts_tablet_manager.cc:1398] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:47.621917 1571 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef: Bootstrap starting.
I20240501 14:00:47.621716 1613 consensus_queue.cc:237] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.621943 1443 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.623628 1571 tablet_bootstrap.cc:654] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.625102 1577 leader_election.cc:288] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.618414 1617 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.630259 1564 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.630487 1564 ts_tablet_manager.cc:1367] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20240501 14:00:47.631500 1564 raft_consensus.cc:348] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.631873 1564 raft_consensus.cc:374] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.631987 1564 raft_consensus.cc:729] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.632284 1564 consensus_queue.cc:260] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.633044 1564 ts_tablet_manager.cc:1398] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240501 14:00:47.633436 1564 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.634549 1571 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef: No bootstrap required, opened a new log
I20240501 14:00:47.634732 1571 ts_tablet_manager.cc:1367] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.006s
I20240501 14:00:47.635322 1564 tablet_bootstrap.cc:654] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.635424 1571 raft_consensus.cc:348] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.635692 1571 raft_consensus.cc:374] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.635772 1571 raft_consensus.cc:729] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.636015 1571 consensus_queue.cc:260] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.637863 1571 ts_tablet_manager.cc:1398] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:00:47.639248 1564 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.639415 1564 ts_tablet_manager.cc:1367] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240501 14:00:47.639338 1267 catalog_manager.cc:5484] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.639838 1267 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.640012 1564 raft_consensus.cc:348] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.640206 1564 raft_consensus.cc:374] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.640326 1564 raft_consensus.cc:729] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.640677 1564 consensus_queue.cc:260] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.641249 1564 ts_tablet_manager.cc:1398] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:00:47.641543 1564 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: Bootstrap starting.
I20240501 14:00:47.641588 1568 raft_consensus.cc:482] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.642236 1568 raft_consensus.cc:504] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.642956 1568 leader_election.cc:288] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.643323 1564 tablet_bootstrap.cc:654] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: Neither blocks nor log segments found. Creating new log.
I20240501 14:00:47.643630 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355ae59edd1c4620904ba23275021582" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef" is_pre_election: true
I20240501 14:00:47.643852 1519 raft_consensus.cc:2454] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 0.
I20240501 14:00:47.644384 1409 leader_election.cc:302] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:47.644450 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355ae59edd1c4620904ba23275021582" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.644778 1369 raft_consensus.cc:2454] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 0.
I20240501 14:00:47.644807 1568 raft_consensus.cc:2781] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.644907 1568 raft_consensus.cc:482] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.645053 1568 raft_consensus.cc:3037] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.645479 1564 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: No bootstrap required, opened a new log
I20240501 14:00:47.645648 1564 ts_tablet_manager.cc:1367] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:00:47.646382 1564 raft_consensus.cc:348] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.646615 1564 raft_consensus.cc:374] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:00:47.646698 1564 raft_consensus.cc:729] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Initialized, Role: FOLLOWER
I20240501 14:00:47.646958 1564 consensus_queue.cc:260] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.647544 1564 ts_tablet_manager.cc:1398] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:47.648048 1568 raft_consensus.cc:504] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.648703 1568 leader_election.cc:288] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:47.648906 1617 raft_consensus.cc:482] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.649045 1617 raft_consensus.cc:504] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.649013 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355ae59edd1c4620904ba23275021582" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:47.649196 1369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355ae59edd1c4620904ba23275021582" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.649283 1519 raft_consensus.cc:3037] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.649334 1369 raft_consensus.cc:3037] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.649830 1617 leader_election.cc:288] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.650133 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.650327 1442 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.650739 1485 leader_election.cc:302] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:47.651037 1617 raft_consensus.cc:2781] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.651132 1617 raft_consensus.cc:482] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.651254 1617 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.651741 1519 raft_consensus.cc:2454] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 1.
I20240501 14:00:47.652206 1369 raft_consensus.cc:2454] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 1.
I20240501 14:00:47.652307 1409 leader_election.cc:302] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:47.652355 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.652524 1370 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.652619 1568 raft_consensus.cc:2781] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.653371 1568 raft_consensus.cc:686] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:47.653693 1568 consensus_queue.cc:237] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.654779 1617 raft_consensus.cc:504] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.655634 1617 leader_election.cc:288] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.655752 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.655961 1370 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.656015 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.656270 1442 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.656587 1268 catalog_manager.cc:5484] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 0 to 1, leader changed from <none> to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 1 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:47.658413 1370 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.658766 1442 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.658797 1482 leader_election.cc:302] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.659044 1617 raft_consensus.cc:2781] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.659224 1617 raft_consensus.cc:686] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:47.659516 1617 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.662034 1268 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.680397 1617 raft_consensus.cc:482] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.680583 1617 raft_consensus.cc:504] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.681236 1617 leader_election.cc:288] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:00:47.681694 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.681699 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.681907 1442 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.681954 1370 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.682284 1485 leader_election.cc:302] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:47.682525 1617 raft_consensus.cc:2781] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.682627 1617 raft_consensus.cc:482] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.682690 1617 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.684574 1617 raft_consensus.cc:504] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.685125 1617 leader_election.cc:288] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:00:47.685545 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.685583 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.685712 1370 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.685786 1442 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.688092 1370 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.688102 1442 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.688498 1482 leader_election.cc:302] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.688794 1617 raft_consensus.cc:2781] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.688941 1617 raft_consensus.cc:686] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:47.689265 1617 consensus_queue.cc:237] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.691887 1268 catalog_manager.cc:5484] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.759243 1617 raft_consensus.cc:482] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.759526 1617 raft_consensus.cc:504] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.760350 1617 leader_election.cc:288] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.760833 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.760936 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.761049 1370 raft_consensus.cc:2454] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.761075 1442 raft_consensus.cc:2454] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.761474 1482 leader_election.cc:302] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.761737 1617 raft_consensus.cc:2781] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.761835 1617 raft_consensus.cc:482] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.761950 1617 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.764567 1617 raft_consensus.cc:504] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.765317 1617 leader_election.cc:288] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.765651 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.765754 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.765856 1370 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.765897 1442 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.768234 1370 raft_consensus.cc:2454] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.768464 1442 raft_consensus.cc:2454] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.768630 1482 leader_election.cc:302] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.768942 1617 raft_consensus.cc:2781] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.769188 1617 raft_consensus.cc:686] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:47.769439 1617 consensus_queue.cc:237] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.771836 1268 catalog_manager.cc:5484] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.775457 1617 raft_consensus.cc:482] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.775568 1617 raft_consensus.cc:504] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.776013 1617 leader_election.cc:288] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.776350 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c36a1a8c09d415d8c1d1860c0d8b622" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.776505 1370 raft_consensus.cc:2454] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.776747 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c36a1a8c09d415d8c1d1860c0d8b622" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.776819 1482 leader_election.cc:302] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:47.776961 1442 raft_consensus.cc:2454] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.777077 1617 raft_consensus.cc:2781] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.777197 1617 raft_consensus.cc:482] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.777284 1617 raft_consensus.cc:3037] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.779042 1617 raft_consensus.cc:504] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.779503 1617 leader_election.cc:288] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.779845 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c36a1a8c09d415d8c1d1860c0d8b622" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.779824 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c36a1a8c09d415d8c1d1860c0d8b622" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.780081 1442 raft_consensus.cc:3037] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.780100 1370 raft_consensus.cc:3037] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.781963 1370 raft_consensus.cc:2454] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.782027 1442 raft_consensus.cc:2454] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.782397 1485 leader_election.cc:302] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:47.782671 1617 raft_consensus.cc:2781] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.782806 1617 raft_consensus.cc:686] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:47.783069 1617 consensus_queue.cc:237] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.785323 1268 catalog_manager.cc:5484] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.893124 1617 raft_consensus.cc:482] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:00:47.893374 1617 raft_consensus.cc:504] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.894048 1617 leader_election.cc:288] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.894565 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed" is_pre_election: true
I20240501 14:00:47.894593 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e" is_pre_election: true
I20240501 14:00:47.894755 1442 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.894788 1370 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 0.
I20240501 14:00:47.895128 1485 leader_election.cc:302] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:47.895383 1617 raft_consensus.cc:2781] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:00:47.895502 1617 raft_consensus.cc:482] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:00:47.895589 1617 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.897622 1617 raft_consensus.cc:504] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.898126 1617 leader_election.cc:288] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:47.898440 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:47.898491 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:47.898603 1370 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.898655 1442 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.899945 1369 raft_consensus.cc:3037] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.900768 1442 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.901058 1370 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 1.
I20240501 14:00:47.901125 1485 leader_election.cc:302] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:47.901413 1617 raft_consensus.cc:2781] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:00:47.901582 1617 raft_consensus.cc:686] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:47.901861 1617 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:47.903293 1568 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.904783 1268 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:47.907279 1633 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.943992 1519 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:00:47.946422 1592 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.950238 1576 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.952996 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:47.960963 1585 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:47.962659 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:00:47.962919 1519 raft_consensus.cc:595] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:00:47.965462 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:00:47.965701 1519 raft_consensus.cc:595] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:47.966670 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:00:47.966893 1519 raft_consensus.cc:595] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:47.968608 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:00:47.968849 1519 raft_consensus.cc:595] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:00:47.969308 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240501 14:00:47.969614 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:47.983574 1592 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240501 14:00:48.007620 1617 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.012562 1613 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:48.035987 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.037216 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.038002 1613 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:48.040319 1494 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.040923 1494 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.048154 1490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.053825 1490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.054597 1490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.057109 1613 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.057791 1370 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 14:00:48.059134 1490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.069296 1490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.081264 1490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.082226 1490 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.084167 1495 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.084854 1495 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.080825 1494 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.088061 1495 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.093997 1613 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.108266 1618 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:48.110139 1498 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.110950 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.137595 1496 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.140170 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.140352 1517 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.140918 1577 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
W20240501 14:00:48.141618 1498 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.148734 1441 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.149389 1613 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.157265 1669 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.167215 1367 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.167667 1618 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.170044 1367 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:00:48.172027 1618 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:48.180615 1498 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.183596 1496 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.183650 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.183769 1669 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.192358 1618 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.194619 1618 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:48.198355 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.205650 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.208250 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.208585 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.212355 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.215221 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.227360 1409 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.194.195:38049, user_credentials={real_user=slave}} blocked reactor thread for 55393.7us
W20240501 14:00:48.229164 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.230320 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.232208 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.233395 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.237840 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.238052 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.241290 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.242691 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.243621 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.244644 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.248659 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.250522 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.297128 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.299376 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.300491 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.326401 1666 raft_consensus.cc:979] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:48.326581 1666 raft_consensus.cc:1067] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
W20240501 14:00:48.328194 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.328322 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.332597 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.335129 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.335311 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.330883 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.340224 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.340793 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.341769 1369 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58886
I20240501 14:00:48.341957 1369 raft_consensus.cc:482] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:48.342085 1369 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:00:48.347328 1664 log.cc:924] Time spent T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
W20240501 14:00:48.351791 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.351910 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.360378 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.366537 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.349412 1690 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
W20240501 14:00:48.428503 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.434353 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.435626 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.436331 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.437333 1485 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.194.194:38379, user_credentials={real_user=slave}} blocked reactor thread for 192728us
I20240501 14:00:48.436626 1369 raft_consensus.cc:504] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:48.441648 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.447187 1666 raft_consensus.cc:979] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:48.447353 1690 raft_consensus.cc:1067] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:48.448338 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:48.448566 1444 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:48.449534 1613 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:00:48.461357 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.465315 1369 leader_election.cc:288] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
W20240501 14:00:48.470829 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.474299 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.465512 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
W20240501 14:00:48.480266 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.476856 1690 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240501 14:00:48.485035 1442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55952
I20240501 14:00:48.485312 1442 raft_consensus.cc:482] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:48.485428 1442 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:00:48.489010 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.489962 1516 raft_consensus.cc:3032] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:48.490072 1516 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
W20240501 14:00:48.490576 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.496737 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.499009 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.502699 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.504091 1516 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:48.507110 1516 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:00:48.513621 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.514267 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.515805 1618 raft_consensus.cc:979] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:48.515921 1618 raft_consensus.cc:1067] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
W20240501 14:00:48.509858 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.526046 1370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58886
I20240501 14:00:48.526259 1370 raft_consensus.cc:482] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:48.526366 1370 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:00:48.531040 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.532608 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.535121 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.537427 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.540318 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.533900 1370 raft_consensus.cc:504] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:48.547520 1444 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:48.548101 1336 leader_election.cc:302] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:48.548648 1702 raft_consensus.cc:979] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:48.548781 1666 raft_consensus.cc:1067] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:48.550906 1445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55952
I20240501 14:00:48.551064 1445 raft_consensus.cc:482] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:48.551156 1445 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:48.549921 1438 raft_consensus.cc:1226] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Rejecting Update request from peer 1e896a0495ca458abad35d883aef4aef for earlier term 1. Current term is 2. Ops: []
W20240501 14:00:48.552235 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.550477 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:48.552845 1443 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:48.556221 1576 raft_consensus.cc:2781] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:48.556360 1576 raft_consensus.cc:686] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:48.558888 1576 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:48.563621 1370 leader_election.cc:288] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
W20240501 14:00:48.564858 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.566484 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:48.556566 1698 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20240501 14:00:48.568527 1517 raft_consensus.cc:3032] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:48.568604 1517 raft_consensus.cc:729] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:48.568886 1517 consensus_queue.cc:260] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:48.569227 1517 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:00:48.574604 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.581861 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.582600 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.583967 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
W20240501 14:00:48.585755 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.586334 1444 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240501 14:00:48.586808 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.588214 1517 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:48.591588 1691 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:00:48.594516 1442 raft_consensus.cc:504] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:48.594745 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.593524 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.589592 1516 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:48.591313 1334 leader_election.cc:302] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:48.596468 1691 raft_consensus.cc:2781] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election won for term 2
W20240501 14:00:48.597976 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.599829 1347 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45550: Service unavailable: leader is not yet ready
I20240501 14:00:48.600329 1442 leader_election.cc:288] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 2 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:48.596608 1691 raft_consensus.cc:686] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:48.600545 1367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:48.600881 1516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:48.601843 1367 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:48.601861 1516 raft_consensus.cc:3032] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:48.601948 1516 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:48.602193 1518 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:00:48.602157 1516 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:48.602449 1516 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:00:48.606209 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.606921 1443 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:48.607278 1438 raft_consensus.cc:1226] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Rejecting Update request from peer 1e896a0495ca458abad35d883aef4aef for earlier term 1. Current term is 2. Ops: [1.1-1.1]
W20240501 14:00:48.608422 1350 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45550: Service unavailable: leader is not yet ready
W20240501 14:00:48.609542 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Service unavailable: leader transfer in progress
I20240501 14:00:48.601275 1691 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:48.610762 1445 raft_consensus.cc:504] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:48.613688 1347 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45550: Service unavailable: leader is not yet ready
I20240501 14:00:48.614956 1367 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 2.
I20240501 14:00:48.615600 1445 leader_election.cc:288] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 2 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:48.615707 1408 leader_election.cc:302] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:48.616591 1674 raft_consensus.cc:2781] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:48.620404 1445 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:00:48.620453 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:48.620605 1519 raft_consensus.cc:3032] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:48.620668 1519 raft_consensus.cc:729] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:48.616710 1674 raft_consensus.cc:686] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:48.621232 1674 consensus_queue.cc:237] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:48.618041 1367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:48.620926 1519 consensus_queue.cc:260] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:48.623207 1519 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:00:48.624433 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
I20240501 14:00:48.621587 1691 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.003s
I20240501 14:00:48.627786 1367 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:48.628679 1518 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240501 14:00:48.629576 1497 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.629577 1493 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.630535 1491 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.632401 1499 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.631927 1496 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
I20240501 14:00:48.633630 1585 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20240501 14:00:48.634107 1687 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
W20240501 14:00:48.634140 1492 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.636458 1496 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45060: Illegal state: replica 1e896a0495ca458abad35d883aef4aef is not leader of this config: current role FOLLOWER
W20240501 14:00:48.643925 1350 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45550: Illegal state: replica c07691f747984bc7a2df956f4f212d8e is not leader of this config: current role FOLLOWER
I20240501 14:00:48.640885 1268 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 1 to 2, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } }
I20240501 14:00:48.651007 1365 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:00:48.650981 1519 raft_consensus.cc:2454] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 2.
W20240501 14:00:48.651991 1349 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45550: Illegal state: replica c07691f747984bc7a2df956f4f212d8e is not leader of this config: current role FOLLOWER
W20240501 14:00:48.652390 1350 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45550: Illegal state: replica c07691f747984bc7a2df956f4f212d8e is not leader of this config: current role FOLLOWER
I20240501 14:00:48.653277 1667 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20240501 14:00:48.654749 1409 leader_election.cc:302] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:48.656059 1667 raft_consensus.cc:2781] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:48.657169 1516 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 2.
I20240501 14:00:48.660777 1667 raft_consensus.cc:686] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
W20240501 14:00:48.662508 1347 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45550: Illegal state: replica c07691f747984bc7a2df956f4f212d8e is not leader of this config: current role FOLLOWER
W20240501 14:00:48.664206 1424 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45462: Service unavailable: leader is not yet ready
I20240501 14:00:48.666225 1515 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240501 14:00:48.667802 1424 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45462: Service unavailable: leader is not yet ready
I20240501 14:00:48.664749 1667 consensus_queue.cc:237] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:48.670647 1421 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45462: Service unavailable: leader is not yet ready
I20240501 14:00:48.671540 1654 mvcc.cc:205] Tried to move back new op lower bound from 7022887111062208512 to 7022887110899965952. Current Snapshot: MvccSnapshot[applied={T|T < 7022887111035293696}]
I20240501 14:00:48.672389 1367 raft_consensus.cc:2454] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 2.
I20240501 14:00:48.673576 1669 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240501 14:00:48.692514 1365 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:00:48.694779 1669 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:48.695205 1518 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:00:48.695889 1669 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:00:48.819806 1261 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 1 to 2, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:48.884517 1267 catalog_manager.cc:5484] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 1 to 2, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 2 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:48.893524 1267 catalog_manager.cc:5484] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 1 to 2, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 2 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:49.649580 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:49.652614 1718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:49.653391 1719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:49.653676 1721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:49.654366 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:49.654642 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:49.665587 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:49.665666 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572049665662 us; error 0 us; skew 500 ppm
I20240501 14:00:49.665839 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:49.667199 30475 webserver.cc:460] Webserver started at http://127.29.194.196:43771/ using document root <none> and password file <none>
I20240501 14:00:49.667418 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:49.667485 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:49.667622 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:49.668241 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-3-root/instance:
uuid: "bf66d98ce8a34944be110a6fa9c75d09"
format_stamp: "Formatted at 2024-05-01 14:00:49 on dist-test-slave-kwk7"
I20240501 14:00:49.671483 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 14:00:49.673120 1726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.673440 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:00:49.673578 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-3-root
uuid: "bf66d98ce8a34944be110a6fa9c75d09"
format_stamp: "Formatted at 2024-05-01 14:00:49 on dist-test-slave-kwk7"
I20240501 14:00:49.673704 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:49.684141 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:49.684634 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:49.685189 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:49.686024 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:49.686138 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.686262 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:49.686311 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.701110 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.196:41675
I20240501 14:00:49.701166 1788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.196:41675 every 8 connection(s)
I20240501 14:00:49.702980 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:49.707068 1793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:49.707816 1794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:49.708971 1796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:49.709161 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:49.709447 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:49.709511 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:49.709560 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572049709557 us; error 0 us; skew 500 ppm
I20240501 14:00:49.709703 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:49.711009 30475 webserver.cc:460] Webserver started at http://127.29.194.197:37919/ using document root <none> and password file <none>
I20240501 14:00:49.711217 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:49.711287 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:49.711413 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:49.712116 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-4-root/instance:
uuid: "b1330e34c41745c1bc55375408448018"
format_stamp: "Formatted at 2024-05-01 14:00:49 on dist-test-slave-kwk7"
I20240501 14:00:49.713301 1789 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37029
I20240501 14:00:49.713425 1789 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:49.713858 1789 heartbeater.cc:511] Master 127.29.194.254:37029 requested a full tablet report, sending...
I20240501 14:00:49.714823 1261 ts_manager.cc:194] Registered new tserver with Master: bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675)
I20240501 14:00:49.715580 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:00:49.715829 1261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35050
I20240501 14:00:49.717084 1801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.717386 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 14:00:49.717481 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-4-root
uuid: "b1330e34c41745c1bc55375408448018"
format_stamp: "Formatted at 2024-05-01 14:00:49 on dist-test-slave-kwk7"
I20240501 14:00:49.717569 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:49.729709 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:49.730237 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:49.730759 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:49.731544 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:49.731621 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.731686 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:49.731721 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.747444 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.197:39411
I20240501 14:00:49.747484 1863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.197:39411 every 8 connection(s)
I20240501 14:00:49.749269 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:00:49.752642 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:49.753281 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:00:49.753860 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:00:49.754770 30475 server_base.cc:1034] running on GCE node
I20240501 14:00:49.755007 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:00:49.755084 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:00:49.755126 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572049755123 us; error 0 us; skew 500 ppm
I20240501 14:00:49.755254 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:00:49.755818 1864 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37029
I20240501 14:00:49.755955 1864 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:49.756358 1864 heartbeater.cc:511] Master 127.29.194.254:37029 requested a full tablet report, sending...
I20240501 14:00:49.756589 30475 webserver.cc:460] Webserver started at http://127.29.194.198:44851/ using document root <none> and password file <none>
I20240501 14:00:49.756803 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:00:49.756870 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:00:49.756980 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:00:49.757222 1261 ts_manager.cc:194] Registered new tserver with Master: b1330e34c41745c1bc55375408448018 (127.29.194.197:39411)
I20240501 14:00:49.757701 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-5-root/instance:
uuid: "8d30ece821d04a59962525029d0af320"
format_stamp: "Formatted at 2024-05-01 14:00:49 on dist-test-slave-kwk7"
I20240501 14:00:49.758122 1261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35058
I20240501 14:00:49.760269 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:00:49.761641 1875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.761931 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:00:49.762039 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-5-root
uuid: "8d30ece821d04a59962525029d0af320"
format_stamp: "Formatted at 2024-05-01 14:00:49 on dist-test-slave-kwk7"
I20240501 14:00:49.762146 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:00:49.775806 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:00:49.776285 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:00:49.776813 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:00:49.777592 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:00:49.777647 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.777756 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:00:49.777810 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:00:49.792702 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.198:35783
I20240501 14:00:49.792778 1937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.198:35783 every 8 connection(s)
I20240501 14:00:49.799670 1938 heartbeater.cc:347] Connected to a master server at 127.29.194.254:37029
I20240501 14:00:49.799818 1938 heartbeater.cc:464] Registering TS with master...
I20240501 14:00:49.800247 1938 heartbeater.cc:511] Master 127.29.194.254:37029 requested a full tablet report, sending...
I20240501 14:00:49.801005 1261 ts_manager.cc:194] Registered new tserver with Master: 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783)
I20240501 14:00:49.801725 1261 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35072
I20240501 14:00:49.970031 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:49.972646 1517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:00:49.972872 1517 raft_consensus.cc:595] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:49.973425 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:00:49.973645 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:49.974867 1666 raft_consensus.cc:979] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:49.975013 1666 raft_consensus.cc:1067] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:49.975576 1367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58886
I20240501 14:00:49.975731 1367 raft_consensus.cc:482] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:49.975826 1367 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:49.977778 1367 raft_consensus.cc:504] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:49.978307 1367 leader_election.cc:288] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:49.978703 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:49.978847 1439 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:49.978819 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:49.979012 1517 raft_consensus.cc:3032] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:49.979081 1517 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:49.979317 1517 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:49.979578 1517 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:49.980729 1439 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:49.981129 1336 leader_election.cc:302] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:49.981421 1576 raft_consensus.cc:2781] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:49.981472 1517 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:49.981593 1576 raft_consensus.cc:686] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:49.981877 1576 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:49.983772 1261 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 1 to 2, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:49.997048 1367 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:49.999120 1439 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:49.999624 1517 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:49.999830 1576 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:50.000380 1715 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:50.002260 1717 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:50.003240 1517 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:50.005286 1439 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
W20240501 14:00:50.006165 1334 consensus_peers.cc:485] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e -> Peer b1330e34c41745c1bc55375408448018 (127.29.194.197:39411): Couldn't send request to peer b1330e34c41745c1bc55375408448018. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae2d33db85414876998a3ed550e44e41. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.008411 1261 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index -1 to 30, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:50.015739 1439 consensus_queue.cc:237] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:00:50.018011 1367 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:50.018523 1674 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:50.020355 1633 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:00:50.020977 1367 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:00:50.026108 1267 catalog_manager.cc:5484] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index -1 to 30, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New cstate: current_term: 2 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:00:50.030308 1439 consensus_queue.cc:237] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:50.030632 1517 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:50.031797 1661 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:50.033409 1514 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
W20240501 14:00:50.036232 1409 consensus_peers.cc:485] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed -> Peer bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675): Couldn't send request to peer bf66d98ce8a34944be110a6fa9c75d09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 348221e7dbd04efcba6185ba29a33a60. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.037572 1367 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:50.038249 1517 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:50.038599 1661 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:50.039350 1661 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:50.041232 1633 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:50.041733 1367 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:50.042412 1517 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:50.046392 1268 catalog_manager.cc:5484] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index -1 to 29, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New cstate: current_term: 1 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:50.046303 1439 consensus_queue.cc:237] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
W20240501 14:00:50.049013 1409 consensus_peers.cc:485] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed -> Peer bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675): Couldn't send request to peer bf66d98ce8a34944be110a6fa9c75d09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 355ae59edd1c4620904ba23275021582. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.049263 1370 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:00:50.049726 1661 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:50.049935 1514 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240501 14:00:50.052016 1408 consensus_peers.cc:485] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46ae855dc1a44159b5f3cca0657dc752. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:50.052872 1633 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:50.053145 1661 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:00:50.053798 1370 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:00:50.058010 1517 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:00:50.058303 1268 catalog_manager.cc:5484] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index -1 to 28, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New cstate: current_term: 1 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:50.059011 1316 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240501 14:00:50.065768 1316 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6ee8e1d3ee3b429ebd15a32afe393e69, TS c07691f747984bc7a2df956f4f212d8e -> TS b1330e34c41745c1bc55375408448018 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:50.065858 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6ee8e1d3ee3b429ebd15a32afe393e69, TS c07691f747984bc7a2df956f4f212d8e -> TS b1330e34c41745c1bc55375408448018 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:50.070781 1316 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 407fed285f274f3c95342a21421b2c66, TS 31d5b9d4a2d2406a972f48086b62ceed -> TS 8d30ece821d04a59962525029d0af320 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:50.070879 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 407fed285f274f3c95342a21421b2c66, TS 31d5b9d4a2d2406a972f48086b62ceed -> TS 8d30ece821d04a59962525029d0af320 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:50.366277 1517 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 14:00:50.366789 1687 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:00:50.372157 1439 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240501 14:00:50.372680 1687 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 14:00:50.481968 1957 ts_tablet_manager.cc:902] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: Initiating tablet copy from peer 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:50.483672 1957 tablet_copy_client.cc:308] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.194:38379
I20240501 14:00:50.489593 1455 tablet_copy_service.cc:140] P 31d5b9d4a2d2406a972f48086b62ceed: Received BeginTabletCopySession request for tablet 355ae59edd1c4620904ba23275021582 from peer bf66d98ce8a34944be110a6fa9c75d09 ({username='slave'} at 127.0.0.1:45484)
I20240501 14:00:50.489717 1455 tablet_copy_service.cc:161] P 31d5b9d4a2d2406a972f48086b62ceed: Beginning new tablet copy session on tablet 355ae59edd1c4620904ba23275021582 from peer bf66d98ce8a34944be110a6fa9c75d09 at {username='slave'} at 127.0.0.1:45484: session id = bf66d98ce8a34944be110a6fa9c75d09-355ae59edd1c4620904ba23275021582
I20240501 14:00:50.491778 1455 tablet_copy_source_session.cc:205] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:50.492779 1957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355ae59edd1c4620904ba23275021582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.494459 1963 ts_tablet_manager.cc:902] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: Initiating tablet copy from peer 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:50.496248 1963 tablet_copy_client.cc:308] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.194:38379
I20240501 14:00:50.498736 1957 tablet_copy_client.cc:783] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:50.499053 1957 tablet_copy_client.cc:647] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:50.500262 1961 tablet_copy_client.cc:1002] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:50.500458 1969 ts_tablet_manager.cc:902] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: Initiating tablet copy from peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:00:50.502985 1455 tablet_copy_service.cc:140] P 31d5b9d4a2d2406a972f48086b62ceed: Received BeginTabletCopySession request for tablet 46ae855dc1a44159b5f3cca0657dc752 from peer 8d30ece821d04a59962525029d0af320 ({username='slave'} at 127.0.0.1:45488)
I20240501 14:00:50.503121 1455 tablet_copy_service.cc:161] P 31d5b9d4a2d2406a972f48086b62ceed: Beginning new tablet copy session on tablet 46ae855dc1a44159b5f3cca0657dc752 from peer 8d30ece821d04a59962525029d0af320 at {username='slave'} at 127.0.0.1:45488: session id = 8d30ece821d04a59962525029d0af320-46ae855dc1a44159b5f3cca0657dc752
I20240501 14:00:50.505622 1969 tablet_copy_client.cc:308] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.193:46821
I20240501 14:00:50.505797 1455 tablet_copy_source_session.cc:205] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:50.507189 1963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ae855dc1a44159b5f3cca0657dc752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.511720 1380 tablet_copy_service.cc:140] P c07691f747984bc7a2df956f4f212d8e: Received BeginTabletCopySession request for tablet ae2d33db85414876998a3ed550e44e41 from peer b1330e34c41745c1bc55375408448018 ({username='slave'} at 127.0.0.1:45572)
I20240501 14:00:50.511888 1380 tablet_copy_service.cc:161] P c07691f747984bc7a2df956f4f212d8e: Beginning new tablet copy session on tablet ae2d33db85414876998a3ed550e44e41 from peer b1330e34c41745c1bc55375408448018 at {username='slave'} at 127.0.0.1:45572: session id = b1330e34c41745c1bc55375408448018-ae2d33db85414876998a3ed550e44e41
I20240501 14:00:50.514712 1380 tablet_copy_source_session.cc:205] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:50.515380 1963 tablet_copy_client.cc:783] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:50.515611 1963 tablet_copy_client.cc:647] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:50.516630 1969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2d33db85414876998a3ed550e44e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.516927 1967 tablet_copy_client.cc:1002] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:50.517314 1976 ts_tablet_manager.cc:902] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: Initiating tablet copy from peer 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:50.524735 1969 tablet_copy_client.cc:783] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:50.525013 1969 tablet_copy_client.cc:647] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:50.526669 1971 tablet_copy_client.cc:1002] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:50.528995 1976 tablet_copy_client.cc:308] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.194:38379
I20240501 14:00:50.529542 1455 tablet_copy_service.cc:140] P 31d5b9d4a2d2406a972f48086b62ceed: Received BeginTabletCopySession request for tablet 348221e7dbd04efcba6185ba29a33a60 from peer bf66d98ce8a34944be110a6fa9c75d09 ({username='slave'} at 127.0.0.1:45484)
I20240501 14:00:50.529650 1455 tablet_copy_service.cc:161] P 31d5b9d4a2d2406a972f48086b62ceed: Beginning new tablet copy session on tablet 348221e7dbd04efcba6185ba29a33a60 from peer bf66d98ce8a34944be110a6fa9c75d09 at {username='slave'} at 127.0.0.1:45484: session id = bf66d98ce8a34944be110a6fa9c75d09-348221e7dbd04efcba6185ba29a33a60
I20240501 14:00:50.531401 1455 tablet_copy_source_session.cc:205] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:50.532295 1976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 348221e7dbd04efcba6185ba29a33a60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:50.536655 1976 tablet_copy_client.cc:783] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:50.536918 1976 tablet_copy_client.cc:647] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:50.538271 1980 tablet_copy_client.cc:1002] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:50.717104 1789 heartbeater.cc:503] Master 127.29.194.254:37029 was elected leader, sending a full tablet report...
I20240501 14:00:50.759196 1864 heartbeater.cc:503] Master 127.29.194.254:37029 was elected leader, sending a full tablet report...
I20240501 14:00:50.802639 1938 heartbeater.cc:503] Master 127.29.194.254:37029 was elected leader, sending a full tablet report...
I20240501 14:00:51.974052 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:51.977955 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 14:00:51.978232 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:52.501327 1957 tablet_copy_client.cc:523] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:52.505851 1957 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap starting.
I20240501 14:00:52.518033 1963 tablet_copy_client.cc:523] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:52.522411 1963 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: Bootstrap starting.
I20240501 14:00:52.527345 1969 tablet_copy_client.cc:523] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:52.531471 1969 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: Bootstrap starting.
I20240501 14:00:52.539568 1976 tablet_copy_client.cc:523] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:52.542325 1976 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap starting.
I20240501 14:00:52.579257 1963 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:52.579706 1963 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: Bootstrap complete.
I20240501 14:00:52.579986 1963 ts_tablet_manager.cc:1367] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: Time spent bootstrapping tablet: real 0.058s user 0.053s sys 0.000s
I20240501 14:00:52.580787 1963 raft_consensus.cc:348] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:52.581043 1963 raft_consensus.cc:729] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d30ece821d04a59962525029d0af320, State: Initialized, Role: LEARNER
I20240501 14:00:52.581254 1963 consensus_queue.cc:260] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:52.588353 1963 ts_tablet_manager.cc:1398] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: Time spent starting tablet: real 0.008s user 0.004s sys 0.005s
I20240501 14:00:52.589193 1969 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=123 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:52.589660 1969 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: Bootstrap complete.
I20240501 14:00:52.589962 1969 ts_tablet_manager.cc:1367] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: Time spent bootstrapping tablet: real 0.059s user 0.041s sys 0.006s
I20240501 14:00:52.591042 1969 raft_consensus.cc:348] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:52.591408 1969 raft_consensus.cc:729] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b1330e34c41745c1bc55375408448018, State: Initialized, Role: LEARNER
I20240501 14:00:52.591665 1969 consensus_queue.cc:260] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:52.593206 1455 tablet_copy_service.cc:342] P 31d5b9d4a2d2406a972f48086b62ceed: Request end of tablet copy session 8d30ece821d04a59962525029d0af320-46ae855dc1a44159b5f3cca0657dc752 received from {username='slave'} at 127.0.0.1:45488
I20240501 14:00:52.593307 1455 tablet_copy_service.cc:433] P 31d5b9d4a2d2406a972f48086b62ceed: ending tablet copy session 8d30ece821d04a59962525029d0af320-46ae855dc1a44159b5f3cca0657dc752 on tablet 46ae855dc1a44159b5f3cca0657dc752 with peer 8d30ece821d04a59962525029d0af320
I20240501 14:00:52.594765 1976 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=128 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:52.595167 1976 tablet_bootstrap.cc:492] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap complete.
I20240501 14:00:52.595412 1976 ts_tablet_manager.cc:1367] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: Time spent bootstrapping tablet: real 0.053s user 0.040s sys 0.011s
I20240501 14:00:52.595548 1957 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:52.595913 1957 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap complete.
I20240501 14:00:52.596154 1957 ts_tablet_manager.cc:1367] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: Time spent bootstrapping tablet: real 0.090s user 0.039s sys 0.004s
I20240501 14:00:52.596293 1976 raft_consensus.cc:348] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:00:52.596566 1976 raft_consensus.cc:729] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: bf66d98ce8a34944be110a6fa9c75d09, State: Initialized, Role: LEARNER
I20240501 14:00:52.596815 1957 raft_consensus.cc:348] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:00:52.597090 1957 raft_consensus.cc:729] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf66d98ce8a34944be110a6fa9c75d09, State: Initialized, Role: LEARNER
I20240501 14:00:52.597265 1957 consensus_queue.cc:260] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:00:52.596828 1976 consensus_queue.cc:260] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:00:52.598338 1976 ts_tablet_manager.cc:1398] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240501 14:00:52.599074 1969 ts_tablet_manager.cc:1398] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20240501 14:00:52.599665 1380 tablet_copy_service.cc:342] P c07691f747984bc7a2df956f4f212d8e: Request end of tablet copy session b1330e34c41745c1bc55375408448018-ae2d33db85414876998a3ed550e44e41 received from {username='slave'} at 127.0.0.1:45572
I20240501 14:00:52.599730 1455 tablet_copy_service.cc:342] P 31d5b9d4a2d2406a972f48086b62ceed: Request end of tablet copy session bf66d98ce8a34944be110a6fa9c75d09-348221e7dbd04efcba6185ba29a33a60 received from {username='slave'} at 127.0.0.1:45484
I20240501 14:00:52.599879 1455 tablet_copy_service.cc:433] P 31d5b9d4a2d2406a972f48086b62ceed: ending tablet copy session bf66d98ce8a34944be110a6fa9c75d09-348221e7dbd04efcba6185ba29a33a60 on tablet 348221e7dbd04efcba6185ba29a33a60 with peer bf66d98ce8a34944be110a6fa9c75d09
I20240501 14:00:52.599763 1380 tablet_copy_service.cc:433] P c07691f747984bc7a2df956f4f212d8e: ending tablet copy session b1330e34c41745c1bc55375408448018-ae2d33db85414876998a3ed550e44e41 on tablet ae2d33db85414876998a3ed550e44e41 with peer b1330e34c41745c1bc55375408448018
I20240501 14:00:52.603981 1957 ts_tablet_manager.cc:1398] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20240501 14:00:52.604651 1455 tablet_copy_service.cc:342] P 31d5b9d4a2d2406a972f48086b62ceed: Request end of tablet copy session bf66d98ce8a34944be110a6fa9c75d09-355ae59edd1c4620904ba23275021582 received from {username='slave'} at 127.0.0.1:45484
I20240501 14:00:52.604771 1455 tablet_copy_service.cc:433] P 31d5b9d4a2d2406a972f48086b62ceed: ending tablet copy session bf66d98ce8a34944be110a6fa9c75d09-355ae59edd1c4620904ba23275021582 on tablet 355ae59edd1c4620904ba23275021582 with peer bf66d98ce8a34944be110a6fa9c75d09
I20240501 14:00:52.702677 1764 raft_consensus.cc:1203] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 14:00:52.821772 1839 raft_consensus.cc:1203] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 14:00:52.835644 1913 raft_consensus.cc:1203] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 14:00:52.870509 1764 raft_consensus.cc:1203] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240501 14:00:53.091264 1716 raft_consensus.cc:1050] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: attempting to promote NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 to VOTER
I20240501 14:00:53.091981 1716 consensus_queue.cc:237] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:53.095300 1367 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:53.095404 1517 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:53.095835 1992 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:53.095300 1764 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEARNER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:53.096343 1992 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:53.096735 1991 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:53.099632 1994 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.100214 1764 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.101482 1517 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.105197 1370 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.106756 1268 catalog_manager.cc:5484] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 30 to 31, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:53.119946 1443 consensus_queue.cc:237] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:53.122454 1764 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:00:53.122619 1370 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:00:53.122936 1994 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:53.123212 1992 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:53.125280 1994 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.125846 1370 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.125874 1764 raft_consensus.cc:2932] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.130261 1267 catalog_manager.cc:5484] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 31 to 32, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New cstate: current_term: 2 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.131315 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 348221e7dbd04efcba6185ba29a33a60 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:53.135449 1508 tablet_service.cc:1514] Processing DeleteTablet for tablet 348221e7dbd04efcba6185ba29a33a60 with delete_type TABLET_DATA_TOMBSTONED (TS 1e896a0495ca458abad35d883aef4aef not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:56882
I20240501 14:00:53.137627 2002 tablet_replica.cc:330] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:00:53.137790 2002 raft_consensus.cc:2229] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.138087 2002 raft_consensus.cc:2258] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.138845 2002 ts_tablet_manager.cc:1875] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:53.143640 2002 ts_tablet_manager.cc:1888] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 14:00:53.143733 2002 log.cc:1196] T 348221e7dbd04efcba6185ba29a33a60 P 1e896a0495ca458abad35d883aef4aef: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root/wals/348221e7dbd04efcba6185ba29a33a60
I20240501 14:00:53.144395 1253 catalog_manager.cc:4830] TS 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): tablet 348221e7dbd04efcba6185ba29a33a60 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:53.230342 2004 raft_consensus.cc:1050] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: attempting to promote NON_VOTER b1330e34c41745c1bc55375408448018 to VOTER
I20240501 14:00:53.231129 2004 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:53.233556 1839 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:53.233733 1444 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:53.234025 2004 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:53.234417 2003 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:53.234486 1517 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:53.235113 1691 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:53.239187 1691 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:53.243291 1443 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:53.244189 1261 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 30 to 31, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:53.247088 1839 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:53.251255 1366 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:53.251206 1517 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:53.253199 1838 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:00:53.253793 1691 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:53.256613 1514 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:00:53.257309 2003 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:53.257300 2004 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:53.257930 1838 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:53.261293 1252 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae2d33db85414876998a3ed550e44e41 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:53.261794 1514 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:53.262475 1261 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 31 to 32, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:53.266647 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2d33db85414876998a3ed550e44e41 with delete_type TABLET_DATA_TOMBSTONED (TS 31d5b9d4a2d2406a972f48086b62ceed not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:55942
I20240501 14:00:53.267355 2009 tablet_replica.cc:330] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:00:53.267601 2009 raft_consensus.cc:2229] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.267910 2009 raft_consensus.cc:2258] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.269650 2009 ts_tablet_manager.cc:1875] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:53.275635 2009 ts_tablet_manager.cc:1888] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 14:00:53.275764 2009 log.cc:1196] T ae2d33db85414876998a3ed550e44e41 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root/wals/ae2d33db85414876998a3ed550e44e41
I20240501 14:00:53.276417 1255 catalog_manager.cc:4830] TS 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379): tablet ae2d33db85414876998a3ed550e44e41 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:53.339617 1991 raft_consensus.cc:1050] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
I20240501 14:00:53.340626 1991 consensus_queue.cc:237] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:53.342864 1366 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:00:53.342872 1514 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:00:53.343474 1913 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:00:53.343737 1992 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:53.344743 1992 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:53.345069 1994 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:53.348029 1992 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:53.349570 1366 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:53.350890 1913 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:53.352506 1514 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:53.355439 1261 catalog_manager.cc:5484] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 29 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:53.360805 1443 consensus_queue.cc:237] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:53.362378 1913 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 14:00:53.362978 1514 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 14:00:53.362953 1716 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:53.363682 1716 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:53.365620 1992 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:53.366113 1913 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:53.366735 1514 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:53.369673 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46ae855dc1a44159b5f3cca0657dc752 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:53.371134 1268 catalog_manager.cc:5484] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 30 to 31, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New cstate: current_term: 1 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:53.374569 1360 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ae855dc1a44159b5f3cca0657dc752 with delete_type TABLET_DATA_TOMBSTONED (TS c07691f747984bc7a2df956f4f212d8e not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:58866
I20240501 14:00:53.375324 2015 tablet_replica.cc:330] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:00:53.375578 2015 raft_consensus.cc:2229] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.375938 2015 raft_consensus.cc:2258] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.376484 1716 raft_consensus.cc:1050] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: attempting to promote NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 to VOTER
I20240501 14:00:53.376987 2015 ts_tablet_manager.cc:1875] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:53.377202 1716 consensus_queue.cc:237] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:53.380113 1514 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:53.380331 1366 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:53.380458 1764 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 LEARNER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:53.380957 1716 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:53.381280 1994 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:53.382028 1992 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:53.383991 2015 ts_tablet_manager.cc:1888] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 14:00:53.384197 2015 log.cc:1196] T 46ae855dc1a44159b5f3cca0657dc752 P c07691f747984bc7a2df956f4f212d8e: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root/wals/46ae855dc1a44159b5f3cca0657dc752
I20240501 14:00:53.385156 1252 catalog_manager.cc:4830] TS c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): tablet 46ae855dc1a44159b5f3cca0657dc752 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:53.391342 1994 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.391948 1366 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.396591 1514 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.396207 1764 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.397760 1259 catalog_manager.cc:5484] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 28 to 29, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:53.403232 1443 consensus_queue.cc:237] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:53.405270 1764 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:00:53.405725 1366 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:00:53.405687 1992 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:53.408924 1992 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:53.409865 1994 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.410310 1764 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.412797 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 355ae59edd1c4620904ba23275021582 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:53.414412 1267 catalog_manager.cc:5484] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 29 to 30, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New cstate: current_term: 1 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:53.415570 1366 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:53.417757 1508 tablet_service.cc:1514] Processing DeleteTablet for tablet 355ae59edd1c4620904ba23275021582 with delete_type TABLET_DATA_TOMBSTONED (TS 1e896a0495ca458abad35d883aef4aef not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:56882
I20240501 14:00:53.418118 2002 tablet_replica.cc:330] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:00:53.418308 2002 raft_consensus.cc:2229] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:53.418578 2002 raft_consensus.cc:2258] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:53.419920 2002 ts_tablet_manager.cc:1875] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:53.424063 2002 ts_tablet_manager.cc:1888] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240501 14:00:53.424149 2002 log.cc:1196] T 355ae59edd1c4620904ba23275021582 P 1e896a0495ca458abad35d883aef4aef: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root/wals/355ae59edd1c4620904ba23275021582
I20240501 14:00:53.424765 1253 catalog_manager.cc:4830] TS 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): tablet 355ae59edd1c4620904ba23275021582 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:53.978626 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:53.989751 1366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:00:53.989926 1366 raft_consensus.cc:595] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:00:53.990752 1366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "407fed285f274f3c95342a21421b2c66"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:00:53.990907 1366 raft_consensus.cc:595] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:00:53.996083 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46ae855dc1a44159b5f3cca0657dc752"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "8d30ece821d04a59962525029d0af320"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:00:53.996234 1444 raft_consensus.cc:595] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Received request to transfer leadership to TS 8d30ece821d04a59962525029d0af320
I20240501 14:00:53.997108 1366 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "b1330e34c41745c1bc55375408448018"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:00:53.997277 1366 raft_consensus.cc:595] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Received request to transfer leadership to TS b1330e34c41745c1bc55375408448018
I20240501 14:00:53.998071 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:00:53.998220 1444 raft_consensus.cc:595] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Received request to transfer leadership to TS 1e896a0495ca458abad35d883aef4aef
I20240501 14:00:53.998581 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 5
I20240501 14:00:53.998792 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:54.014871 1691 raft_consensus.cc:979] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:54.015033 1691 raft_consensus.cc:1067] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:54.015554 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "407fed285f274f3c95342a21421b2c66"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55950
I20240501 14:00:54.015718 1444 raft_consensus.cc:482] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:54.015831 1444 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.017802 1444 raft_consensus.cc:504] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.018360 1444 leader_election.cc:288] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 2 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:54.018725 1366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:54.018779 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:54.018911 1366 raft_consensus.cc:3032] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:54.018975 1514 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.018999 1366 raft_consensus.cc:729] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:54.019222 1366 consensus_queue.cc:260] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.019512 1366 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.020807 1514 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 2.
I20240501 14:00:54.021160 1409 leader_election.cc:302] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:54.021340 1366 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 2.
I20240501 14:00:54.021358 1991 raft_consensus.cc:2781] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:54.021785 1991 raft_consensus.cc:686] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:54.022053 1991 consensus_queue.cc:237] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.024444 1268 catalog_manager.cc:5484] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 1 to 2, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 2 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:54.207811 2004 raft_consensus.cc:979] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower b1330e34c41745c1bc55375408448018 to start an election
I20240501 14:00:54.207996 2004 raft_consensus.cc:1067] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Signalling peer b1330e34c41745c1bc55375408448018 to start an election
I20240501 14:00:54.208536 1838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41"
dest_uuid: "b1330e34c41745c1bc55375408448018"
from {username='slave'} at 127.0.0.1:44020
I20240501 14:00:54.208738 1838 raft_consensus.cc:482] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:54.208851 1838 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:54.210960 1838 raft_consensus.cc:504] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:54.211661 1838 leader_election.cc:288] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [CANDIDATE]: Term 3 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:54.212116 1366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "b1330e34c41745c1bc55375408448018" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:54.212311 1366 raft_consensus.cc:3032] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:00:54.212404 1366 raft_consensus.cc:729] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:54.212651 1366 consensus_queue.cc:260] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:54.213001 1366 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:54.215024 1366 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1330e34c41745c1bc55375408448018 in term 3.
I20240501 14:00:54.215461 1802 leader_election.cc:302] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1330e34c41745c1bc55375408448018, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:54.216030 2030 raft_consensus.cc:2781] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:00:54.216423 2030 raft_consensus.cc:686] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 LEADER]: Becoming Leader. State: Replica: b1330e34c41745c1bc55375408448018, State: Running, Role: LEADER
I20240501 14:00:54.216761 2030 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:54.218919 1268 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 reported cstate change: term changed from 2 to 3, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to b1330e34c41745c1bc55375408448018 (127.29.194.197). New cstate: current_term: 3 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:54.219781 2023 raft_consensus.cc:979] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower 8d30ece821d04a59962525029d0af320 to start an election
I20240501 14:00:54.219929 2023 raft_consensus.cc:1067] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Signalling peer 8d30ece821d04a59962525029d0af320 to start an election
I20240501 14:00:54.220614 1913 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46ae855dc1a44159b5f3cca0657dc752"
dest_uuid: "8d30ece821d04a59962525029d0af320"
from {username='slave'} at 127.0.0.1:38320
I20240501 14:00:54.220813 1913 raft_consensus.cc:482] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:54.220937 1913 raft_consensus.cc:3037] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.221441 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "b1330e34c41745c1bc55375408448018" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:54.221624 1514 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:54.223573 1913 raft_consensus.cc:504] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:54.224184 1514 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1330e34c41745c1bc55375408448018 in term 3.
I20240501 14:00:54.224834 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ae855dc1a44159b5f3cca0657dc752" candidate_uuid: "8d30ece821d04a59962525029d0af320" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:54.225023 1444 raft_consensus.cc:3032] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:54.225116 1444 raft_consensus.cc:729] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:54.225356 1444 consensus_queue.cc:260] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:54.225667 1444 raft_consensus.cc:3037] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.225729 1913 leader_election.cc:288] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [CANDIDATE]: Term 2 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:54.227947 1444 raft_consensus.cc:2454] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d30ece821d04a59962525029d0af320 in term 2.
I20240501 14:00:54.228299 1879 leader_election.cc:302] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, 8d30ece821d04a59962525029d0af320; no voters:
I20240501 14:00:54.228900 2033 raft_consensus.cc:2781] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:54.229444 2033 raft_consensus.cc:686] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 2 LEADER]: Becoming Leader. State: Replica: 8d30ece821d04a59962525029d0af320, State: Running, Role: LEADER
I20240501 14:00:54.229693 2033 consensus_queue.cc:237] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:54.230098 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46ae855dc1a44159b5f3cca0657dc752" candidate_uuid: "8d30ece821d04a59962525029d0af320" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:54.230296 1514 raft_consensus.cc:3037] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.232010 1268 catalog_manager.cc:5484] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 reported cstate change: term changed from 1 to 2, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to 8d30ece821d04a59962525029d0af320 (127.29.194.198). New cstate: current_term: 2 leader_uuid: "8d30ece821d04a59962525029d0af320" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:54.232427 1514 raft_consensus.cc:2454] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d30ece821d04a59962525029d0af320 in term 2.
I20240501 14:00:54.274307 2004 raft_consensus.cc:979] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:54.274466 2004 raft_consensus.cc:1067] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:54.275034 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55950
I20240501 14:00:54.275223 1444 raft_consensus.cc:482] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:54.275333 1444 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.277211 1444 raft_consensus.cc:504] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.277814 1444 leader_election.cc:288] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 2 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:54.278263 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:54.278228 1366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:54.278486 1514 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.278483 1366 raft_consensus.cc:3032] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:54.278681 1366 raft_consensus.cc:729] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:54.278961 1366 consensus_queue.cc:260] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.279348 1366 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:54.281533 1514 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 2.
I20240501 14:00:54.281937 1409 leader_election.cc:302] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:54.282032 1366 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 2.
I20240501 14:00:54.282202 1991 raft_consensus.cc:2781] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:54.282352 1991 raft_consensus.cc:686] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:54.282646 1991 consensus_queue.cc:237] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.285208 1268 catalog_manager.cc:5484] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 1 to 2, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 2 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:54.416849 2023 raft_consensus.cc:979] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:00:54.417070 2023 raft_consensus.cc:1067] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Signalling peer 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:00:54.417697 1514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:56890
I20240501 14:00:54.417980 1514 raft_consensus.cc:482] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:54.418118 1514 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:54.420918 1514 raft_consensus.cc:504] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.421610 1514 leader_election.cc:288] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 3 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:00:54.421952 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:54.422020 1366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:54.422201 1444 raft_consensus.cc:3032] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:00:54.422199 1366 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:54.422335 1444 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:54.422618 1444 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.423014 1444 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:54.425237 1366 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 3.
I20240501 14:00:54.425616 1482 leader_election.cc:302] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:54.425808 1444 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 3.
I20240501 14:00:54.425920 1666 raft_consensus.cc:2781] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:00:54.426393 1666 raft_consensus.cc:686] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:54.426625 1514 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:54.426674 1666 consensus_queue.cc:237] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.429107 1366 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:54.429301 1444 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:54.429714 1268 catalog_manager.cc:5484] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 2 to 3, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 3 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:54.430971 2040 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:54.431583 2037 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:54.434389 2040 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:54.435032 1366 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:54.437706 1444 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:54.440716 1268 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index -1 to 29, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:54.441426 1366 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
W20240501 14:00:54.442552 1483 consensus_peers.cc:485] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef -> Peer b1330e34c41745c1bc55375408448018 (127.29.194.197:39411): Couldn't send request to peer b1330e34c41745c1bc55375408448018. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78e839ad73564640b24fe9efc9f41f08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:54.449620 1514 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:54.449636 1444 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:54.450394 1691 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:54.450780 2004 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:54.453886 2004 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
W20240501 14:00:54.455752 1333 consensus_peers.cc:485] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 916a3f31691044f783d655695bf0a369. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:54.455670 1514 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:54.459491 1517 consensus_queue.cc:237] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:00:54.460274 1267 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index -1 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:54.469069 1444 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:00:54.469571 2037 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:00:54.470722 1370 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 14:00:54.471549 2037 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:00:54.471479 1443 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index -1 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:54.479635 1666 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:00:54.480270 1444 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
W20240501 14:00:54.480926 1483 consensus_peers.cc:485] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef -> Peer bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675): Couldn't send request to peer bf66d98ce8a34944be110a6fa9c75d09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ca80506974245cb887559f7f884fb98. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:54.481678 1370 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:00:54.485337 1259 catalog_manager.cc:5484] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index -1 to 27, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:54.486166 1517 consensus_queue.cc:237] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:54.488621 1444 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:00:54.489917 1666 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:54.488719 1370 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:00:54.491274 1666 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:54.494001 2040 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:54.494328 1444 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
W20240501 14:00:54.496398 1482 consensus_peers.cc:485] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c36a1a8c09d415d8c1d1860c0d8b622. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:54.497577 1370 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:54.498889 1268 catalog_manager.cc:5484] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index -1 to 28, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:54.499909 1366 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:54.502036 1444 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20240501 14:00:54.502507 1334 consensus_peers.cc:485] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e -> Peer b1330e34c41745c1bc55375408448018 (127.29.194.197:39411): Couldn't send request to peer b1330e34c41745c1bc55375408448018. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ee8e1d3ee3b429ebd15a32afe393e69. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:54.503116 2050 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:00:54.504387 1517 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 14:00:54.504892 2050 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 14:00:54.506835 1691 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:54.507397 1444 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:54.508147 1517 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
W20240501 14:00:54.513088 1316 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20240501 14:00:54.512904 1267 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index -1 to 27, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:00:54.520058 1370 raft_consensus.cc:1261] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:00:54.520541 2023 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 14:00:54.520671 1316 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d3b31989cdae4e9ab72ad37c9e61f903, TS 1e896a0495ca458abad35d883aef4aef -> TS b1330e34c41745c1bc55375408448018 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:54.520771 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d3b31989cdae4e9ab72ad37c9e61f903, TS 1e896a0495ca458abad35d883aef4aef -> TS b1330e34c41745c1bc55375408448018 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:54.525619 1517 raft_consensus.cc:1261] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:00:54.527091 2023 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:54.637909 1370 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 14:00:54.638442 2030 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:00:54.643889 1517 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 14:00:54.645025 2031 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:00:54.716065 1514 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:00:54.716585 2023 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:54.719980 1366 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:00:54.720614 2023 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:54.722407 1514 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:00:54.722937 2033 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:54.739207 1444 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:00:54.739913 2034 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:54.843168 2077 ts_tablet_manager.cc:902] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: Initiating tablet copy from peer 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:54.844945 2077 tablet_copy_client.cc:308] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.195:38049
I20240501 14:00:54.852358 1529 tablet_copy_service.cc:140] P 1e896a0495ca458abad35d883aef4aef: Received BeginTabletCopySession request for tablet 7ca80506974245cb887559f7f884fb98 from peer bf66d98ce8a34944be110a6fa9c75d09 ({username='slave'} at 127.0.0.1:45076)
I20240501 14:00:54.852555 1529 tablet_copy_service.cc:161] P 1e896a0495ca458abad35d883aef4aef: Beginning new tablet copy session on tablet 7ca80506974245cb887559f7f884fb98 from peer bf66d98ce8a34944be110a6fa9c75d09 at {username='slave'} at 127.0.0.1:45076: session id = bf66d98ce8a34944be110a6fa9c75d09-7ca80506974245cb887559f7f884fb98
I20240501 14:00:54.853225 2083 ts_tablet_manager.cc:902] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: Initiating tablet copy from peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:00:54.855540 1529 tablet_copy_source_session.cc:205] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:54.856484 2083 tablet_copy_client.cc:308] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.193:46821
I20240501 14:00:54.857735 2077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca80506974245cb887559f7f884fb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.863173 2077 tablet_copy_client.cc:783] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:54.863472 2077 tablet_copy_client.cc:647] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:54.865094 2081 tablet_copy_client.cc:1002] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:54.874378 1380 tablet_copy_service.cc:140] P c07691f747984bc7a2df956f4f212d8e: Received BeginTabletCopySession request for tablet 916a3f31691044f783d655695bf0a369 from peer 8d30ece821d04a59962525029d0af320 ({username='slave'} at 127.0.0.1:45594)
I20240501 14:00:54.874560 1380 tablet_copy_service.cc:161] P c07691f747984bc7a2df956f4f212d8e: Beginning new tablet copy session on tablet 916a3f31691044f783d655695bf0a369 from peer 8d30ece821d04a59962525029d0af320 at {username='slave'} at 127.0.0.1:45594: session id = 8d30ece821d04a59962525029d0af320-916a3f31691044f783d655695bf0a369
I20240501 14:00:54.876734 1380 tablet_copy_source_session.cc:205] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:54.878336 2083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916a3f31691044f783d655695bf0a369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.883677 2083 tablet_copy_client.cc:783] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:54.883908 2083 tablet_copy_client.cc:647] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:54.885370 2087 tablet_copy_client.cc:1002] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:54.947871 1444 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:00:54.948426 2037 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:54.953014 1370 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:00:54.953678 2040 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:54.960178 2092 ts_tablet_manager.cc:902] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: Initiating tablet copy from peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:00:54.965643 2092 tablet_copy_client.cc:308] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.193:46821
I20240501 14:00:54.966197 1380 tablet_copy_service.cc:140] P c07691f747984bc7a2df956f4f212d8e: Received BeginTabletCopySession request for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 from peer b1330e34c41745c1bc55375408448018 ({username='slave'} at 127.0.0.1:45572)
I20240501 14:00:54.966334 1380 tablet_copy_service.cc:161] P c07691f747984bc7a2df956f4f212d8e: Beginning new tablet copy session on tablet 6ee8e1d3ee3b429ebd15a32afe393e69 from peer b1330e34c41745c1bc55375408448018 at {username='slave'} at 127.0.0.1:45572: session id = b1330e34c41745c1bc55375408448018-6ee8e1d3ee3b429ebd15a32afe393e69
I20240501 14:00:54.969592 1380 tablet_copy_source_session.cc:205] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:54.970839 2092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee8e1d3ee3b429ebd15a32afe393e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.973129 2097 ts_tablet_manager.cc:902] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: Initiating tablet copy from peer 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:54.977402 2092 tablet_copy_client.cc:783] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:54.977672 2092 tablet_copy_client.cc:647] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:54.978533 2097 tablet_copy_client.cc:308] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.195:38049
I20240501 14:00:54.979177 1529 tablet_copy_service.cc:140] P 1e896a0495ca458abad35d883aef4aef: Received BeginTabletCopySession request for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 from peer 8d30ece821d04a59962525029d0af320 ({username='slave'} at 127.0.0.1:45064)
I20240501 14:00:54.979331 1529 tablet_copy_service.cc:161] P 1e896a0495ca458abad35d883aef4aef: Beginning new tablet copy session on tablet 8c36a1a8c09d415d8c1d1860c0d8b622 from peer 8d30ece821d04a59962525029d0af320 at {username='slave'} at 127.0.0.1:45064: session id = 8d30ece821d04a59962525029d0af320-8c36a1a8c09d415d8c1d1860c0d8b622
I20240501 14:00:54.979568 2096 tablet_copy_client.cc:1002] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:54.982035 1529 tablet_copy_source_session.cc:205] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:54.983031 2097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c36a1a8c09d415d8c1d1860c0d8b622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:54.988871 2097 tablet_copy_client.cc:783] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:54.989154 2097 tablet_copy_client.cc:647] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:54.990675 2101 tablet_copy_client.cc:1002] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:55.049760 2102 ts_tablet_manager.cc:902] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: Initiating tablet copy from peer 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:55.051328 2102 tablet_copy_client.cc:308] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.195:38049
I20240501 14:00:55.051894 1529 tablet_copy_service.cc:140] P 1e896a0495ca458abad35d883aef4aef: Received BeginTabletCopySession request for tablet 78e839ad73564640b24fe9efc9f41f08 from peer b1330e34c41745c1bc55375408448018 ({username='slave'} at 127.0.0.1:45062)
I20240501 14:00:55.052016 1529 tablet_copy_service.cc:161] P 1e896a0495ca458abad35d883aef4aef: Beginning new tablet copy session on tablet 78e839ad73564640b24fe9efc9f41f08 from peer b1330e34c41745c1bc55375408448018 at {username='slave'} at 127.0.0.1:45062: session id = b1330e34c41745c1bc55375408448018-78e839ad73564640b24fe9efc9f41f08
I20240501 14:00:55.054286 1529 tablet_copy_source_session.cc:205] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:55.055294 2102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e839ad73564640b24fe9efc9f41f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:55.060775 2102 tablet_copy_client.cc:783] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:55.060977 2102 tablet_copy_client.cc:647] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:55.062310 2106 tablet_copy_client.cc:1002] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: tablet copy: Injecting latency into file download: 2000
I20240501 14:00:55.999105 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:56.003032 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:00:56.003255 1443 raft_consensus.cc:595] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Received request to transfer leadership to TS 1e896a0495ca458abad35d883aef4aef
I20240501 14:00:56.004405 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "355ae59edd1c4620904ba23275021582"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "bf66d98ce8a34944be110a6fa9c75d09"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:00:56.004565 1443 raft_consensus.cc:595] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Received request to transfer leadership to TS bf66d98ce8a34944be110a6fa9c75d09
I20240501 14:00:56.005420 1514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:00:56.005543 1514 raft_consensus.cc:595] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:56.006419 1514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c36a1a8c09d415d8c1d1860c0d8b622"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:00:56.006551 1514 raft_consensus.cc:595] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:00:56.006925 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240501 14:00:56.007156 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:56.101696 1666 raft_consensus.cc:979] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:56.101848 2040 raft_consensus.cc:1067] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:56.102440 1370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58886
I20240501 14:00:56.102629 1370 raft_consensus.cc:482] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:56.102747 1370 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.105373 1370 raft_consensus.cc:504] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:56.105989 1370 leader_election.cc:288] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:56.106364 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:56.106534 1514 raft_consensus.cc:3032] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:56.106616 1514 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:56.106842 1514 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:56.107344 1514 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.109611 1514 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:56.110195 1334 leader_election.cc:302] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:56.110443 2108 raft_consensus.cc:2781] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:56.110574 2108 raft_consensus.cc:686] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:56.110824 2108 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:56.119019 1259 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 1 to 2, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:56.106295 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:56.129335 1443 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.130553 1370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58866
I20240501 14:00:56.130755 1370 raft_consensus.cc:595] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Received request to transfer leadership
I20240501 14:00:56.131686 1443 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:56.136489 1370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58866
I20240501 14:00:56.136637 1370 raft_consensus.cc:595] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Received request to transfer leadership
W20240501 14:00:56.136962 1252 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 78e839ad73564640b24fe9efc9f41f08 already in progress [suppressed 6 similar messages]
I20240501 14:00:56.137774 1370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58866
I20240501 14:00:56.137887 1370 raft_consensus.cc:595] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Received request to transfer leadership
W20240501 14:00:56.138245 1316 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 78e839ad73564640b24fe9efc9f41f08 already in progress
W20240501 14:00:56.138310 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 78e839ad73564640b24fe9efc9f41f08 already in progress
I20240501 14:00:56.202672 1991 raft_consensus.cc:979] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:00:56.202842 1991 raft_consensus.cc:1067] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Signalling peer 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:00:56.203377 1517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:56890
I20240501 14:00:56.203543 1517 raft_consensus.cc:482] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:56.203641 1517 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:56.205754 1517 raft_consensus.cc:504] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:56.206339 1517 leader_election.cc:288] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 3 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:56.206739 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:56.206900 1370 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:56.206912 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:56.207131 1443 raft_consensus.cc:3032] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:00:56.207208 1443 raft_consensus.cc:729] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:56.207434 1443 consensus_queue.cc:260] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:56.207784 1443 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:56.209120 1370 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 3.
I20240501 14:00:56.209586 1482 leader_election.cc:302] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:56.209832 2040 raft_consensus.cc:2781] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:00:56.209964 2040 raft_consensus.cc:686] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:56.210204 1443 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 3.
I20240501 14:00:56.210235 2040 consensus_queue.cc:237] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:56.212783 1259 catalog_manager.cc:5484] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 2 to 3, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 3 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:00:56.304576 2116 raft_consensus.cc:979] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower bf66d98ce8a34944be110a6fa9c75d09 to start an election
I20240501 14:00:56.304761 2116 raft_consensus.cc:1067] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Signalling peer bf66d98ce8a34944be110a6fa9c75d09 to start an election
I20240501 14:00:56.305331 1764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "355ae59edd1c4620904ba23275021582"
dest_uuid: "bf66d98ce8a34944be110a6fa9c75d09"
from {username='slave'} at 127.0.0.1:55636
I20240501 14:00:56.305528 1764 raft_consensus.cc:482] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:56.305706 1764 raft_consensus.cc:3037] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.308161 1764 raft_consensus.cc:504] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:56.309371 1764 leader_election.cc:288] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [CANDIDATE]: Term 2 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:00:56.309350 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355ae59edd1c4620904ba23275021582" candidate_uuid: "bf66d98ce8a34944be110a6fa9c75d09" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:56.309571 1443 raft_consensus.cc:3032] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:56.309639 1443 raft_consensus.cc:729] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:56.309900 1443 consensus_queue.cc:260] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:56.310308 1443 raft_consensus.cc:3037] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.313688 1443 raft_consensus.cc:2454] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf66d98ce8a34944be110a6fa9c75d09 in term 2.
I20240501 14:00:56.314072 1730 leader_election.cc:302] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, bf66d98ce8a34944be110a6fa9c75d09; no voters:
I20240501 14:00:56.314780 2120 raft_consensus.cc:2781] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:56.317636 2120 raft_consensus.cc:686] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEADER]: Becoming Leader. State: Replica: bf66d98ce8a34944be110a6fa9c75d09, State: Running, Role: LEADER
I20240501 14:00:56.317756 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "355ae59edd1c4620904ba23275021582" candidate_uuid: "bf66d98ce8a34944be110a6fa9c75d09" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:56.318030 1370 raft_consensus.cc:3037] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.318195 2120 consensus_queue.cc:237] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:56.320991 1370 raft_consensus.cc:2454] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf66d98ce8a34944be110a6fa9c75d09 in term 2.
I20240501 14:00:56.324797 1259 catalog_manager.cc:5484] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 reported cstate change: term changed from 1 to 2, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196). New cstate: current_term: 2 leader_uuid: "bf66d98ce8a34944be110a6fa9c75d09" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:56.374719 1666 raft_consensus.cc:979] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:56.374890 2040 raft_consensus.cc:1067] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:56.375556 1370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c36a1a8c09d415d8c1d1860c0d8b622"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58886
I20240501 14:00:56.375763 1370 raft_consensus.cc:482] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:56.375857 1370 raft_consensus.cc:3037] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.378341 1370 raft_consensus.cc:504] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:56.379175 1370 leader_election.cc:288] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:56.379725 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c36a1a8c09d415d8c1d1860c0d8b622" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:56.379935 1443 raft_consensus.cc:3037] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.380906 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c36a1a8c09d415d8c1d1860c0d8b622" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:56.381053 1517 raft_consensus.cc:3032] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:00:56.381125 1517 raft_consensus.cc:729] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:56.381368 1517 consensus_queue.cc:260] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:56.381722 1517 raft_consensus.cc:3037] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:00:56.383416 1443 raft_consensus.cc:2454] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:56.383531 1517 raft_consensus.cc:2454] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 2.
I20240501 14:00:56.383914 1336 leader_election.cc:302] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:56.384173 2108 raft_consensus.cc:2781] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:00:56.384363 2108 raft_consensus.cc:686] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:56.384662 2108 consensus_queue.cc:237] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:56.388770 1259 catalog_manager.cc:5484] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 1 to 2, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:56.604707 1334 consensus_peers.cc:485] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e -> Peer b1330e34c41745c1bc55375408448018 (127.29.194.197:39411): Couldn't send request to peer b1330e34c41745c1bc55375408448018. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:56.613857 1443 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:56.617118 2108 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:56.620064 2108 raft_consensus.cc:979] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:56.620211 2108 raft_consensus.cc:1067] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:56.621073 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55950
I20240501 14:00:56.621291 1444 raft_consensus.cc:482] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:56.621398 1444 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:56.624934 1444 raft_consensus.cc:504] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:56.625941 1443 raft_consensus.cc:1226] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Rejecting Update request from peer c07691f747984bc7a2df956f4f212d8e for earlier term 2. Current term is 3. Ops: []
I20240501 14:00:56.626827 2109 consensus_queue.cc:1046] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: INVALID_TERM, Last received: 2.30, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:56.627979 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:56.628480 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:56.628659 1517 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:56.632352 1517 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 3.
I20240501 14:00:56.632992 1409 leader_election.cc:302] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:00:56.633318 1991 raft_consensus.cc:2781] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:00:56.633457 1991 raft_consensus.cc:686] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:56.634092 1991 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:56.633383 1444 leader_election.cc:288] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 3 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:56.636682 1370 raft_consensus.cc:3032] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:00:56.636834 1370 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:56.637151 1370 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:56.637331 1259 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 2 to 3, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 3 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:56.638536 1517 raft_consensus.cc:1226] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Rejecting Update request from peer c07691f747984bc7a2df956f4f212d8e for earlier term 2. Current term is 3. Ops: []
I20240501 14:00:56.637655 1370 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:56.643633 1370 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 3.
I20240501 14:00:56.703171 1370 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:56.703725 2120 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:56.710578 1444 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:56.712174 2130 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:56.813654 1444 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:00:56.814332 2040 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:56.816377 1517 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:00:56.817008 2108 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:56.820732 1367 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:00:56.821379 2134 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:56.822454 1443 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:00:56.823145 2109 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240501 14:00:56.827517 1333 consensus_peers.cc:485] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:56.867349 2077 tablet_copy_client.cc:523] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:56.870834 2077 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap starting.
I20240501 14:00:56.886013 2083 tablet_copy_client.cc:523] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:56.888511 2083 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: Bootstrap starting.
I20240501 14:00:56.906559 2077 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:56.907007 2077 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap complete.
I20240501 14:00:56.907261 2077 ts_tablet_manager.cc:1367] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: Time spent bootstrapping tablet: real 0.036s user 0.031s sys 0.004s
I20240501 14:00:56.907838 2077 raft_consensus.cc:348] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:00:56.908106 2077 raft_consensus.cc:729] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf66d98ce8a34944be110a6fa9c75d09, State: Initialized, Role: LEARNER
I20240501 14:00:56.908313 2077 consensus_queue.cc:260] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:00:56.908907 2077 ts_tablet_manager.cc:1398] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:56.909660 1529 tablet_copy_service.cc:342] P 1e896a0495ca458abad35d883aef4aef: Request end of tablet copy session bf66d98ce8a34944be110a6fa9c75d09-7ca80506974245cb887559f7f884fb98 received from {username='slave'} at 127.0.0.1:45076
I20240501 14:00:56.909761 1529 tablet_copy_service.cc:433] P 1e896a0495ca458abad35d883aef4aef: ending tablet copy session bf66d98ce8a34944be110a6fa9c75d09-7ca80506974245cb887559f7f884fb98 on tablet 7ca80506974245cb887559f7f884fb98 with peer bf66d98ce8a34944be110a6fa9c75d09
I20240501 14:00:56.926848 2083 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:56.927254 2083 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: Bootstrap complete.
I20240501 14:00:56.927510 2083 ts_tablet_manager.cc:1367] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: Time spent bootstrapping tablet: real 0.039s user 0.030s sys 0.007s
I20240501 14:00:56.928153 2083 raft_consensus.cc:348] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:56.928333 2083 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8d30ece821d04a59962525029d0af320, State: Initialized, Role: LEARNER
I20240501 14:00:56.928545 2083 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:56.929145 2083 ts_tablet_manager.cc:1398] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:00:56.929716 1380 tablet_copy_service.cc:342] P c07691f747984bc7a2df956f4f212d8e: Request end of tablet copy session 8d30ece821d04a59962525029d0af320-916a3f31691044f783d655695bf0a369 received from {username='slave'} at 127.0.0.1:45594
I20240501 14:00:56.929803 1380 tablet_copy_service.cc:433] P c07691f747984bc7a2df956f4f212d8e: ending tablet copy session 8d30ece821d04a59962525029d0af320-916a3f31691044f783d655695bf0a369 on tablet 916a3f31691044f783d655695bf0a369 with peer 8d30ece821d04a59962525029d0af320
I20240501 14:00:56.980471 2092 tablet_copy_client.cc:523] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:56.983392 2092 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: Bootstrap starting.
I20240501 14:00:56.991863 2097 tablet_copy_client.cc:523] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:56.995180 2097 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: Bootstrap starting.
I20240501 14:00:57.016386 2092 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:57.016777 2092 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: Bootstrap complete.
I20240501 14:00:57.017001 2092 ts_tablet_manager.cc:1367] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: Time spent bootstrapping tablet: real 0.034s user 0.023s sys 0.013s
I20240501 14:00:57.017664 2092 raft_consensus.cc:348] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:57.017931 2092 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b1330e34c41745c1bc55375408448018, State: Initialized, Role: LEARNER
I20240501 14:00:57.018110 2092 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:57.019106 2092 ts_tablet_manager.cc:1398] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:00:57.019613 1380 tablet_copy_service.cc:342] P c07691f747984bc7a2df956f4f212d8e: Request end of tablet copy session b1330e34c41745c1bc55375408448018-6ee8e1d3ee3b429ebd15a32afe393e69 received from {username='slave'} at 127.0.0.1:45572
I20240501 14:00:57.019696 1380 tablet_copy_service.cc:433] P c07691f747984bc7a2df956f4f212d8e: ending tablet copy session b1330e34c41745c1bc55375408448018-6ee8e1d3ee3b429ebd15a32afe393e69 on tablet 6ee8e1d3ee3b429ebd15a32afe393e69 with peer b1330e34c41745c1bc55375408448018
I20240501 14:00:57.032765 2097 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:57.033159 2097 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: Bootstrap complete.
I20240501 14:00:57.033396 2097 ts_tablet_manager.cc:1367] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: Time spent bootstrapping tablet: real 0.038s user 0.034s sys 0.005s
I20240501 14:00:57.034027 2097 raft_consensus.cc:348] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:57.034209 2097 raft_consensus.cc:729] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d30ece821d04a59962525029d0af320, State: Initialized, Role: LEARNER
I20240501 14:00:57.034410 2097 consensus_queue.cc:260] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:57.034904 2097 ts_tablet_manager.cc:1398] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:00:57.035537 1529 tablet_copy_service.cc:342] P 1e896a0495ca458abad35d883aef4aef: Request end of tablet copy session 8d30ece821d04a59962525029d0af320-8c36a1a8c09d415d8c1d1860c0d8b622 received from {username='slave'} at 127.0.0.1:45064
I20240501 14:00:57.035642 1529 tablet_copy_service.cc:433] P 1e896a0495ca458abad35d883aef4aef: ending tablet copy session 8d30ece821d04a59962525029d0af320-8c36a1a8c09d415d8c1d1860c0d8b622 on tablet 8c36a1a8c09d415d8c1d1860c0d8b622 with peer 8d30ece821d04a59962525029d0af320
I20240501 14:00:57.062834 2102 tablet_copy_client.cc:523] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:00:57.065901 2102 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: Bootstrap starting.
I20240501 14:00:57.101927 2102 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:00:57.102375 2102 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: Bootstrap complete.
I20240501 14:00:57.102654 2102 ts_tablet_manager.cc:1367] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: Time spent bootstrapping tablet: real 0.037s user 0.028s sys 0.008s
I20240501 14:00:57.103277 2102 raft_consensus.cc:348] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:57.103541 2102 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b1330e34c41745c1bc55375408448018, State: Initialized, Role: LEARNER
I20240501 14:00:57.103747 2102 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:00:57.104241 2102 ts_tablet_manager.cc:1398] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:00:57.105254 1529 tablet_copy_service.cc:342] P 1e896a0495ca458abad35d883aef4aef: Request end of tablet copy session b1330e34c41745c1bc55375408448018-78e839ad73564640b24fe9efc9f41f08 received from {username='slave'} at 127.0.0.1:45062
I20240501 14:00:57.105377 1529 tablet_copy_service.cc:433] P 1e896a0495ca458abad35d883aef4aef: ending tablet copy session b1330e34c41745c1bc55375408448018-78e839ad73564640b24fe9efc9f41f08 on tablet 78e839ad73564640b24fe9efc9f41f08 with peer b1330e34c41745c1bc55375408448018
I20240501 14:00:57.128816 1838 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 1 LEARNER]: Advancing to term 3
I20240501 14:00:57.131561 1838 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 3 LEARNER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:00:57.132051 1991 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:57.134537 1991 raft_consensus.cc:1050] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: attempting to promote NON_VOTER b1330e34c41745c1bc55375408448018 to VOTER
W20240501 14:00:57.134645 1991 raft_consensus.cc:1054] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: Unable to promote non-voter b1330e34c41745c1bc55375408448018: Illegal state: Leader has not yet committed an operation in its own term
I20240501 14:00:57.138844 1517 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:00:57.139290 1991 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:57.144649 1367 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:00:57.146847 2147 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:57.147423 2149 raft_consensus.cc:1050] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: attempting to promote NON_VOTER b1330e34c41745c1bc55375408448018 to VOTER
I20240501 14:00:57.149999 2149 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:57.152267 1517 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 14:00:57.152364 1838 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 3 LEARNER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 14:00:57.152876 2147 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:57.153067 1991 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:57.153285 1367 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 14:00:57.153782 1991 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:57.157132 1991 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 29 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.158031 1517 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.158558 1370 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.161805 1838 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 29 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.164144 1268 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index 29 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.170202 1443 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:57.172082 1517 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 14:00:57.172082 1838 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 14:00:57.172735 1991 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:00:57.173002 2147 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:00:57.174845 1991 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.175916 1838 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.178493 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78e839ad73564640b24fe9efc9f41f08 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:57.179237 1517 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.180140 1261 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 32 to 33, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New cstate: current_term: 3 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:57.183621 1360 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e839ad73564640b24fe9efc9f41f08 with delete_type TABLET_DATA_TOMBSTONED (TS c07691f747984bc7a2df956f4f212d8e not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:58866
I20240501 14:00:57.185321 2156 tablet_replica.cc:330] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:00:57.185717 2156 raft_consensus.cc:2229] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.186092 2156 raft_consensus.cc:2258] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.186856 2156 ts_tablet_manager.cc:1875] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:57.192984 2156 ts_tablet_manager.cc:1888] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20240501 14:00:57.193099 2156 log.cc:1196] T 78e839ad73564640b24fe9efc9f41f08 P c07691f747984bc7a2df956f4f212d8e: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root/wals/78e839ad73564640b24fe9efc9f41f08
I20240501 14:00:57.193693 1252 catalog_manager.cc:4830] TS c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): tablet 78e839ad73564640b24fe9efc9f41f08 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:57.230461 1913 raft_consensus.cc:3037] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Advancing to term 2
I20240501 14:00:57.347635 1764 raft_consensus.cc:1203] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240501 14:00:57.366370 1838 raft_consensus.cc:1203] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20240501 14:00:57.457296 1913 raft_consensus.cc:1203] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20240501 14:00:57.626148 2161 raft_consensus.cc:1050] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
I20240501 14:00:57.627008 2161 consensus_queue.cc:237] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:57.629087 1913 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 LEARNER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:57.629125 1517 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:57.629653 1444 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:57.630175 2163 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:57.630455 2162 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:57.631601 2161 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:57.633805 2163 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:57.634366 1444 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:57.635075 1913 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:57.639703 1517 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:57.640203 1261 catalog_manager.cc:5484] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 28 to 30, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:57.644783 1370 consensus_queue.cc:237] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:57.646287 1913 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:57.646834 2161 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:57.648365 1517 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 14:00:57.648784 2163 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:57.649771 1912 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:57.649170 2162 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:57.650290 1517 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:57.655198 1259 catalog_manager.cc:5484] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 reported cstate change: config changed from index 30 to 31, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:57.655828 1252 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:57.659356 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 with delete_type TABLET_DATA_TOMBSTONED (TS 31d5b9d4a2d2406a972f48086b62ceed not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:55942
I20240501 14:00:57.660063 2164 tablet_replica.cc:330] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:00:57.660282 2164 raft_consensus.cc:2229] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.660635 2164 raft_consensus.cc:2258] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.661495 2164 ts_tablet_manager.cc:1875] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:57.667277 2164 ts_tablet_manager.cc:1888] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.30
I20240501 14:00:57.667387 2164 log.cc:1196] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root/wals/8c36a1a8c09d415d8c1d1860c0d8b622
I20240501 14:00:57.668183 1255 catalog_manager.cc:4830] TS 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379): tablet 8c36a1a8c09d415d8c1d1860c0d8b622 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:57.744963 2163 raft_consensus.cc:1050] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: attempting to promote NON_VOTER b1330e34c41745c1bc55375408448018 to VOTER
I20240501 14:00:57.745750 2163 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 25, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:57.748050 1838 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:00:57.748090 1517 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:00:57.748733 1444 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 14:00:57.748745 2162 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:57.749171 2161 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:57.749399 2127 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:57.753762 2162 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.754820 1517 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.755077 1838 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.759764 1261 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index 27 to 28, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.763784 1444 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.766229 1367 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:57.768494 1444 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:00:57.768515 1838 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 14:00:57.769042 2162 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:57.769222 2127 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:57.770934 2162 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.771595 1444 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.771811 1838 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:00:57.775008 1252 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:57.776455 1267 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 28 to 29, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:57.781292 1508 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 with delete_type TABLET_DATA_TOMBSTONED (TS 1e896a0495ca458abad35d883aef4aef not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:56882
I20240501 14:00:57.781908 2169 tablet_replica.cc:330] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:00:57.782095 2169 raft_consensus.cc:2229] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.782343 2169 raft_consensus.cc:2258] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.783308 2169 ts_tablet_manager.cc:1875] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:57.788904 2169 ts_tablet_manager.cc:1888] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20240501 14:00:57.788993 2169 log.cc:1196] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 1e896a0495ca458abad35d883aef4aef: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root/wals/6ee8e1d3ee3b429ebd15a32afe393e69
I20240501 14:00:57.789537 1253 catalog_manager.cc:4830] TS 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): tablet 6ee8e1d3ee3b429ebd15a32afe393e69 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:57.806061 1666 raft_consensus.cc:1050] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: attempting to promote NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 to VOTER
I20240501 14:00:57.806830 1666 consensus_queue.cc:237] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:57.809206 1444 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:00:57.809293 1764 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 LEARNER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:00:57.810093 2040 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:57.810231 1367 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 14:00:57.810751 2040 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:57.811748 2040 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 14:00:57.822885 2172 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:57.823408 1764 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:57.828192 1444 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:57.828481 1261 catalog_manager.cc:5484] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index 27 to 28, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:57.831295 1370 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:57.836572 1517 consensus_queue.cc:237] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:00:57.838229 1764 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:57.838514 1444 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 14:00:57.838773 2172 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:57.839008 1666 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:00:57.841146 2172 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:57.841850 1444 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:57.841943 1764 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:57.845314 1267 catalog_manager.cc:5484] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 28 to 29, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:00:57.845683 1253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ca80506974245cb887559f7f884fb98 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:57.848862 1360 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca80506974245cb887559f7f884fb98 with delete_type TABLET_DATA_TOMBSTONED (TS c07691f747984bc7a2df956f4f212d8e not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:58866
I20240501 14:00:57.849704 2176 tablet_replica.cc:330] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:00:57.849893 2176 raft_consensus.cc:2229] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:57.850062 2176 raft_consensus.cc:2258] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:57.850772 2176 ts_tablet_manager.cc:1875] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:57.855392 2176 ts_tablet_manager.cc:1888] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240501 14:00:57.855475 2176 log.cc:1196] T 7ca80506974245cb887559f7f884fb98 P c07691f747984bc7a2df956f4f212d8e: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root/wals/7ca80506974245cb887559f7f884fb98
I20240501 14:00:57.856040 1252 catalog_manager.cc:4830] TS c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): tablet 7ca80506974245cb887559f7f884fb98 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:58.007480 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:00:58.011080 1514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:00:58.011299 1514 raft_consensus.cc:595] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:00:58.012252 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:00:58.012431 1444 raft_consensus.cc:595] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Received request to transfer leadership to TS 1e896a0495ca458abad35d883aef4aef
I20240501 14:00:58.013365 1367 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:00:58.013517 1367 raft_consensus.cc:595] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:00:58.013924 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 14:00:58.014111 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:00:58.035132 2170 raft_consensus.cc:979] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:58.035295 2170 raft_consensus.cc:1067] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:58.035809 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55952
I20240501 14:00:58.035988 1444 raft_consensus.cc:482] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:58.036088 1444 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.037954 1444 raft_consensus.cc:504] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:58.038549 1444 leader_election.cc:288] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 4 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:58.038836 1367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:58.039055 1367 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.039036 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bdda4e923dc4caaba534f1e8b5693e2" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:58.039212 1514 raft_consensus.cc:3032] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:00:58.039275 1514 raft_consensus.cc:729] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:58.039506 1514 consensus_queue.cc:260] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:58.039848 1514 raft_consensus.cc:3037] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.041600 1367 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 4.
I20240501 14:00:58.041951 1408 leader_election.cc:302] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:58.042191 1991 raft_consensus.cc:2781] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:00:58.042258 1514 raft_consensus.cc:2454] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 4.
I20240501 14:00:58.042567 1991 raft_consensus.cc:686] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:58.042824 1991 consensus_queue.cc:237] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:00:58.045579 2161 raft_consensus.cc:1050] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
W20240501 14:00:58.045694 2161 raft_consensus.cc:1054] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Unable to promote non-voter 8d30ece821d04a59962525029d0af320: Service unavailable: leader transfer in progress
I20240501 14:00:58.045610 1261 catalog_manager.cc:5484] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 3 to 4, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 4 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:58.245792 2161 raft_consensus.cc:979] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:58.245963 2161 raft_consensus.cc:1067] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:00:58.246479 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55950
I20240501 14:00:58.246670 1444 raft_consensus.cc:482] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:58.246759 1444 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:58.249269 1444 raft_consensus.cc:504] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:58.249895 1444 leader_election.cc:288] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 3 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:58.250155 1367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:00:58.250301 1367 raft_consensus.cc:3032] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:00:58.250365 1367 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:58.250612 1367 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:58.250955 1367 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:58.252591 1367 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 3.
I20240501 14:00:58.252930 1408 leader_election.cc:302] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:58.253238 1991 raft_consensus.cc:2781] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:00:58.253294 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:58.253367 1991 raft_consensus.cc:686] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:58.253474 1514 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:00:58.253732 1991 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:58.255806 1514 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 3.
I20240501 14:00:58.256605 1261 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 2 to 3, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 3 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:58.260838 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55942
I20240501 14:00:58.260983 1444 raft_consensus.cc:595] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Received request to transfer leadership
I20240501 14:00:58.262485 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55942
I20240501 14:00:58.262616 1444 raft_consensus.cc:595] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Received request to transfer leadership
I20240501 14:00:58.264237 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55942
I20240501 14:00:58.264364 1444 raft_consensus.cc:595] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Received request to transfer leadership
W20240501 14:00:58.264778 1316 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 916a3f31691044f783d655695bf0a369 already in progress
W20240501 14:00:58.264837 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 916a3f31691044f783d655695bf0a369 already in progress
I20240501 14:00:58.290436 2177 raft_consensus.cc:979] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:00:58.290553 2177 raft_consensus.cc:1067] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Signalling peer 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:00:58.291110 1514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:56890
I20240501 14:00:58.291250 1514 raft_consensus.cc:482] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:58.291332 1514 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.293172 1514 raft_consensus.cc:504] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:58.293874 1514 leader_election.cc:288] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 4 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), b1330e34c41745c1bc55375408448018 (127.29.194.197:39411)
I20240501 14:00:58.294091 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:58.294164 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: true dest_uuid: "b1330e34c41745c1bc55375408448018"
I20240501 14:00:58.294270 1444 raft_consensus.cc:3032] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:00:58.294328 1838 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.294348 1444 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:58.294564 1444 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:58.294829 1444 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.296190 1838 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 4.
I20240501 14:00:58.296548 1483 leader_election.cc:302] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, b1330e34c41745c1bc55375408448018; no voters:
I20240501 14:00:58.296777 1444 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 4.
I20240501 14:00:58.296880 2170 raft_consensus.cc:2781] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:00:58.297010 2170 raft_consensus.cc:686] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 4 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:00:58.297245 2170 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:00:58.299572 1261 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 3 to 4, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 4 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:58.432449 1370 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 14:00:58.433060 2177 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:58.437479 1514 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 14:00:58.438139 1991 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:58.649591 1912 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 2 LEARNER]: Advancing to term 3
I20240501 14:00:58.651728 1912 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 3 LEARNER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:00:58.652269 1991 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:58.654497 1991 raft_consensus.cc:1050] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
W20240501 14:00:58.654615 1991 raft_consensus.cc:1054] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: Unable to promote non-voter 8d30ece821d04a59962525029d0af320: Service unavailable: leader transfer in progress
I20240501 14:00:58.687139 1444 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240501 14:00:58.687641 2170 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:00:58.691035 1838 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240501 14:00:58.692184 2172 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:00:58.777726 1370 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:00:58.778223 1991 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:58.780277 2177 raft_consensus.cc:979] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:58.780423 1991 raft_consensus.cc:1067] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:00:58.781116 1367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58870
I20240501 14:00:58.781301 1367 raft_consensus.cc:482] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:00:58.781391 1367 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.781745 1514 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:00:58.782151 2177 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:00:58.785023 1367 raft_consensus.cc:504] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:58.785454 1370 raft_consensus.cc:1226] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Rejecting Update request from peer 31d5b9d4a2d2406a972f48086b62ceed for earlier term 3. Current term is 4. Ops: []
I20240501 14:00:58.787010 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:00:58.787185 1444 raft_consensus.cc:3032] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:00:58.787268 1444 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:00:58.787592 1444 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:58.788008 1444 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.791643 1444 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 4.
I20240501 14:00:58.792065 1336 leader_election.cc:302] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:00:58.792310 2161 raft_consensus.cc:2781] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:00:58.793839 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:00:58.794044 1514 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:00:58.796833 1367 leader_election.cc:288] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 4 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:00:58.797986 1514 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 4.
I20240501 14:00:58.798853 2161 raft_consensus.cc:686] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:00:58.799130 2161 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:58.801446 1259 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 3 to 4, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 4 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:00:59.237299 1912 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 3 LEARNER]: Advancing to term 4
I20240501 14:00:59.239827 1912 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 4 LEARNER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:00:59.240239 2161 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:59.241953 2161 raft_consensus.cc:1050] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
W20240501 14:00:59.242067 2161 raft_consensus.cc:1054] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Unable to promote non-voter 8d30ece821d04a59962525029d0af320: Illegal state: Leader has not yet committed an operation in its own term
I20240501 14:00:59.271049 1367 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:00:59.273365 1838 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 14:00:59.273469 1444 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
W20240501 14:00:59.273548 1333 consensus_peers.cc:485] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ee8e1d3ee3b429ebd15a32afe393e69. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:00:59.273921 2161 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:59.274209 2201 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:00:59.277129 2161 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:59.277658 1838 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:59.281229 1444 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:00:59.281745 1259 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 29 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:00:59.282378 1316 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20240501 14:00:59.286461 1316 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 916a3f31691044f783d655695bf0a369, TS 1e896a0495ca458abad35d883aef4aef -> TS bf66d98ce8a34944be110a6fa9c75d09 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
W20240501 14:00:59.286526 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 916a3f31691044f783d655695bf0a369, TS 1e896a0495ca458abad35d883aef4aef -> TS bf66d98ce8a34944be110a6fa9c75d09 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
W20240501 14:00:59.288875 1316 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d3b31989cdae4e9ab72ad37c9e61f903, TS 31d5b9d4a2d2406a972f48086b62ceed -> TS b1330e34c41745c1bc55375408448018 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:59.288937 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d3b31989cdae4e9ab72ad37c9e61f903, TS 31d5b9d4a2d2406a972f48086b62ceed -> TS b1330e34c41745c1bc55375408448018 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
W20240501 14:00:59.291123 1316 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 348221e7dbd04efcba6185ba29a33a60, TS c07691f747984bc7a2df956f4f212d8e -> TS 8d30ece821d04a59962525029d0af320 move failed, destination replica disappeared from tablet's Raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
W20240501 14:00:59.291183 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 348221e7dbd04efcba6185ba29a33a60, TS c07691f747984bc7a2df956f4f212d8e -> TS 8d30ece821d04a59962525029d0af320 move failed, destination replica disappeared from tablet's Raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
W20240501 14:00:59.304312 1316 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ae2d33db85414876998a3ed550e44e41, TS 1e896a0495ca458abad35d883aef4aef -> TS bf66d98ce8a34944be110a6fa9c75d09 move failed, destination replica disappeared from tablet's Raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
W20240501 14:00:59.304384 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ae2d33db85414876998a3ed550e44e41, TS 1e896a0495ca458abad35d883aef4aef -> TS bf66d98ce8a34944be110a6fa9c75d09 move failed, destination replica disappeared from tablet's Raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
W20240501 14:00:59.310739 1316 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 46ae855dc1a44159b5f3cca0657dc752, TS 31d5b9d4a2d2406a972f48086b62ceed -> TS b1330e34c41745c1bc55375408448018 move failed, destination replica disappeared from tablet's Raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
W20240501 14:00:59.310827 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 46ae855dc1a44159b5f3cca0657dc752, TS 31d5b9d4a2d2406a972f48086b62ceed -> TS b1330e34c41745c1bc55375408448018 move failed, destination replica disappeared from tablet's Raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:59.375093 1514 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:00:59.375567 2196 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:59.380481 1444 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:00:59.381817 2161 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:00:59.382259 2196 raft_consensus.cc:1050] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
I20240501 14:00:59.382944 2196 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 31, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:59.386123 1912 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 4 LEARNER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 14:00:59.386538 1514 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 14:00:59.386580 2210 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:00:59.386932 1443 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 14:00:59.387248 2201 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:00:59.387414 2210 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:00:59.389868 2161 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:59.391736 1514 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:59.393376 1912 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:59.395340 1261 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:00:59.397955 1443 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:59.401547 1367 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:00:59.403626 1912 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240501 14:00:59.404316 2201 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:00:59.404860 1514 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240501 14:00:59.405256 2161 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:00:59.407130 2210 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:59.407940 1912 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:59.408057 1514 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:59.411764 1252 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 916a3f31691044f783d655695bf0a369 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:00:59.412124 1261 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 reported cstate change: config changed from index 33 to 34, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New cstate: current_term: 4 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:00:59.418409 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 916a3f31691044f783d655695bf0a369 with delete_type TABLET_DATA_TOMBSTONED (TS 31d5b9d4a2d2406a972f48086b62ceed not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:55942
I20240501 14:00:59.424260 2212 tablet_replica.cc:330] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:00:59.424434 2212 raft_consensus.cc:2229] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 14:00:59.424633 2212 raft_consensus.cc:2258] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:00:59.425331 2212 ts_tablet_manager.cc:1875] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:00:59.430646 2212 ts_tablet_manager.cc:1888] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20240501 14:00:59.430758 2212 log.cc:1196] T 916a3f31691044f783d655695bf0a369 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root/wals/916a3f31691044f783d655695bf0a369
I20240501 14:00:59.431353 1255 catalog_manager.cc:4830] TS 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379): tablet 916a3f31691044f783d655695bf0a369 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:00:59.659708 2213 ts_tablet_manager.cc:902] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: Initiating tablet copy from peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:00:59.660939 2213 tablet_copy_client.cc:308] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.193:46821
I20240501 14:00:59.661486 1380 tablet_copy_service.cc:140] P c07691f747984bc7a2df956f4f212d8e: Received BeginTabletCopySession request for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 from peer 8d30ece821d04a59962525029d0af320 ({username='slave'} at 127.0.0.1:45594)
I20240501 14:00:59.661605 1380 tablet_copy_service.cc:161] P c07691f747984bc7a2df956f4f212d8e: Beginning new tablet copy session on tablet 6ee8e1d3ee3b429ebd15a32afe393e69 from peer 8d30ece821d04a59962525029d0af320 at {username='slave'} at 127.0.0.1:45594: session id = 8d30ece821d04a59962525029d0af320-6ee8e1d3ee3b429ebd15a32afe393e69
I20240501 14:00:59.664098 1380 tablet_copy_source_session.cc:205] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:00:59.665150 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee8e1d3ee3b429ebd15a32afe393e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:00:59.670392 2213 tablet_copy_client.cc:783] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 0 data blocks...
I20240501 14:00:59.670581 2213 tablet_copy_client.cc:647] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 1 WAL segments...
I20240501 14:00:59.671954 2217 tablet_copy_client.cc:1002] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:00.014503 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:00.017643 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:01:00.017838 1443 raft_consensus.cc:595] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:01:00.018796 1367 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:01:00.018963 1367 raft_consensus.cc:595] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:01:00.019846 1514 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:01:00.019979 1514 raft_consensus.cc:595] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 4 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:01:00.020355 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 14:01:00.020571 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:00.196260 2193 raft_consensus.cc:979] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:00.196427 2219 raft_consensus.cc:1067] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:00.197077 1367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58870
I20240501 14:01:00.197234 1367 raft_consensus.cc:482] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:00.197328 1367 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:00.199353 1367 raft_consensus.cc:504] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:00.200083 1367 leader_election.cc:288] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 3 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675)
I20240501 14:01:00.200376 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:01:00.200554 1443 raft_consensus.cc:3032] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:00.200628 1443 raft_consensus.cc:729] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:00.201162 1443 consensus_queue.cc:260] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:00.201454 1443 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:00.203233 1443 raft_consensus.cc:2454] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 3.
I20240501 14:01:00.203552 1336 leader_election.cc:302] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:01:00.203827 2218 raft_consensus.cc:2781] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:00.204051 2218 raft_consensus.cc:686] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 3 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:00.204434 2218 consensus_queue.cc:237] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:00.207592 1259 catalog_manager.cc:5484] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 2 to 3, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 3 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.220465 1764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "348221e7dbd04efcba6185ba29a33a60" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "bf66d98ce8a34944be110a6fa9c75d09"
I20240501 14:01:00.220630 1764 raft_consensus.cc:3037] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:00.222363 1764 raft_consensus.cc:2454] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 3.
I20240501 14:01:00.286784 2218 raft_consensus.cc:979] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:01:00.286957 2218 raft_consensus.cc:1067] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:01:00.287618 1443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55950
I20240501 14:01:00.287904 1443 raft_consensus.cc:482] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:00.288028 1443 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:00.290831 1443 raft_consensus.cc:504] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:00.291536 1443 leader_election.cc:288] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 3 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), b1330e34c41745c1bc55375408448018 (127.29.194.197:39411)
I20240501 14:01:00.291965 1367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:01:00.292124 1367 raft_consensus.cc:3032] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:00.292183 1367 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:00.292543 1367 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:00.292974 1367 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:00.293323 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "b1330e34c41745c1bc55375408448018"
I20240501 14:01:00.293490 1838 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:00.295194 1367 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 3.
I20240501 14:01:00.295308 1838 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 3.
I20240501 14:01:00.295627 1408 leader_election.cc:302] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:01:00.295933 2219 raft_consensus.cc:2781] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:00.296052 2219 raft_consensus.cc:686] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:00.296247 2219 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:00.299017 1259 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 2 to 3, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 3 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.302676 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55942
I20240501 14:01:00.302799 1443 raft_consensus.cc:595] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Received request to transfer leadership
I20240501 14:01:00.304389 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55942
I20240501 14:01:00.304503 1443 raft_consensus.cc:595] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Received request to transfer leadership
I20240501 14:01:00.305595 1443 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55942
I20240501 14:01:00.305702 1443 raft_consensus.cc:595] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Received request to transfer leadership
W20240501 14:01:00.306074 1316 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 6ee8e1d3ee3b429ebd15a32afe393e69 already in progress
W20240501 14:01:00.306138 1316 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 6ee8e1d3ee3b429ebd15a32afe393e69 already in progress
I20240501 14:01:00.367866 2172 raft_consensus.cc:979] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:01:00.368017 2172 raft_consensus.cc:1067] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 4 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:01:00.368597 1443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55952
I20240501 14:01:00.368786 1443 raft_consensus.cc:482] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:00.368887 1443 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:00.371140 1443 raft_consensus.cc:504] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:00.371659 1443 leader_election.cc:288] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 5 election: Requested vote from peers 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049), b1330e34c41745c1bc55375408448018 (127.29.194.197:39411)
I20240501 14:01:00.372027 1514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:01:00.372038 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "b1330e34c41745c1bc55375408448018"
I20240501 14:01:00.372184 1514 raft_consensus.cc:3032] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:00.372236 1838 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:00.372246 1514 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:01:00.372562 1514 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:00.372891 1514 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:00.374258 1838 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 5.
I20240501 14:01:00.374601 1409 leader_election.cc:302] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, b1330e34c41745c1bc55375408448018; no voters:
I20240501 14:01:00.374840 2219 raft_consensus.cc:2781] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:00.374857 1514 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 5.
I20240501 14:01:00.374998 2219 raft_consensus.cc:686] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:00.375244 2219 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:00.377527 1259 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 4 to 5, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 5 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.587954 1443 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 14:01:00.588500 2210 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:00.592571 1764 raft_consensus.cc:1261] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 14:01:00.593020 2218 consensus_queue.cc:1035] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:00.755071 1514 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240501 14:01:00.755578 2219 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:00.758672 1838 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240501 14:01:00.759286 2236 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
W20240501 14:01:00.818169 1408 consensus_peers.cc:485] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:00.839286 1838 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:01:00.839797 2193 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:00.841859 2236 raft_consensus.cc:979] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower b1330e34c41745c1bc55375408448018 to start an election
I20240501 14:01:00.842005 2219 raft_consensus.cc:1067] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Signalling peer b1330e34c41745c1bc55375408448018 to start an election
I20240501 14:01:00.842562 1839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "b1330e34c41745c1bc55375408448018"
from {username='slave'} at 127.0.0.1:44040
I20240501 14:01:00.842764 1839 raft_consensus.cc:482] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:00.842861 1839 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:00.843312 1370 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:01:00.843770 2219 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:00.845571 1839 raft_consensus.cc:504] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:00.846719 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "b1330e34c41745c1bc55375408448018" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:01:00.846927 1370 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:00.847548 1839 leader_election.cc:288] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [CANDIDATE]: Term 4 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:01:00.849493 1370 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1330e34c41745c1bc55375408448018 in term 4.
I20240501 14:01:00.849830 1802 leader_election.cc:302] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1330e34c41745c1bc55375408448018, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:01:00.850049 2151 raft_consensus.cc:2781] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:00.850414 2151 raft_consensus.cc:686] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 LEADER]: Becoming Leader. State: Replica: b1330e34c41745c1bc55375408448018, State: Running, Role: LEADER
I20240501 14:01:00.850654 2151 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:00.852653 1267 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 reported cstate change: term changed from 3 to 4, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to b1330e34c41745c1bc55375408448018 (127.29.194.197). New cstate: current_term: 4 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:00.853111 1443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "b1330e34c41745c1bc55375408448018" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:01:00.853240 1443 raft_consensus.cc:3032] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:00.853307 1443 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:00.853528 1443 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:00.853819 1443 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:00.855909 1443 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1330e34c41745c1bc55375408448018 in term 4.
I20240501 14:01:01.311568 1443 consensus_queue.cc:237] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:01.313573 1370 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240501 14:01:01.313573 1514 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
W20240501 14:01:01.313802 1409 consensus_peers.cc:485] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed -> Peer b1330e34c41745c1bc55375408448018 (127.29.194.197:39411): Couldn't send request to peer b1330e34c41745c1bc55375408448018. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bdda4e923dc4caaba534f1e8b5693e2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:01.314327 2236 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:01.314628 2219 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:01.317055 2236 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index -1 to 31, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:01.317677 1370 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index -1 to 31, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:01.318039 1514 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index -1 to 31, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:01.322249 1517 consensus_queue.cc:237] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:01.322166 1267 catalog_manager.cc:5484] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index -1 to 31, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New cstate: current_term: 4 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:01.324633 1764 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
W20240501 14:01:01.325038 1482 consensus_peers.cc:485] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ca80506974245cb887559f7f884fb98. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:01.325178 2172 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:01:01.326525 1443 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 14:01:01.327366 2249 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:01:01.328637 2172 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:01:01.330363 1443 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:01:01.332032 1764 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:01:01.333755 1261 catalog_manager.cc:5484] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 29 to 30, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:01:01.336644 1839 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:01:01.340015 1370 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240501 14:01:01.340585 1517 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20240501 14:01:01.341131 2151 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:01.341387 2255 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:01.344009 2244 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:01.344635 1370 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:01.350800 1259 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 32 to 34, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New cstate: current_term: 3 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:01.363731 1370 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:01:01.364311 2244 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240501 14:01:01.365039 1803 consensus_peers.cc:485] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 -> Peer bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675): Couldn't send request to peer bf66d98ce8a34944be110a6fa9c75d09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae2d33db85414876998a3ed550e44e41. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:01.372829 1514 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 32 to 34, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:01.374814 1443 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:01:01.375381 2151 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:01.379237 1764 consensus_queue.cc:237] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:01.382704 1367 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:01:01.383409 2160 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:01.386010 2160 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:01.390203 1370 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:01.391337 1444 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
W20240501 14:01:01.394006 1408 consensus_peers.cc:485] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78e839ad73564640b24fe9efc9f41f08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:01.394194 1517 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240501 14:01:01.394685 2219 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 14:01:01.395665 1261 catalog_manager.cc:5484] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 30 to 32, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New cstate: current_term: 2 leader_uuid: "bf66d98ce8a34944be110a6fa9c75d09" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:01.396076 1444 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:01:01.396709 2267 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:01.397784 2236 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Committing config change with OpId 5.36: config changed from index 33 to 36, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:01:01.398382 1517 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 33 to 36, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:01:01.404071 1261 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index 33 to 36, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New cstate: current_term: 5 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:01:01.405026 1912 consensus_queue.cc:237] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:01:01.411146 1439 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:01:01.411695 2124 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:01.413112 1839 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240501 14:01:01.413942 2236 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 14:01:01.416226 1839 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 33 to 36, NON_VOTER 8d30ece821d04a59962525029d0af320 (127.29.194.198) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } } }
I20240501 14:01:01.423206 1444 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
W20240501 14:01:01.424016 1802 consensus_peers.cc:485] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 -> Peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783): Couldn't send request to peer 8d30ece821d04a59962525029d0af320. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:01:01.425202 1728 consensus_peers.cc:485] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 -> Peer b1330e34c41745c1bc55375408448018 (127.29.194.197:39411): Couldn't send request to peer b1330e34c41745c1bc55375408448018. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 355ae59edd1c4620904ba23275021582. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:01.428465 1517 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:01:01.429028 2124 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:01.432035 2276 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:01.432951 1444 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
W20240501 14:01:01.433444 1877 consensus_peers.cc:485] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 -> Peer bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675): Couldn't send request to peer bf66d98ce8a34944be110a6fa9c75d09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46ae855dc1a44159b5f3cca0657dc752. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:01.437047 1261 catalog_manager.cc:5484] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 31 to 33, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New cstate: current_term: 2 leader_uuid: "8d30ece821d04a59962525029d0af320" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:01.441423 1514 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:01.672297 2213 tablet_copy_client.cc:523] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:01.675554 2213 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: Bootstrap starting.
I20240501 14:01:01.693003 2279 ts_tablet_manager.cc:902] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: Initiating tablet copy from peer 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:01:01.701932 2279 tablet_copy_client.cc:308] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.194:38379
I20240501 14:01:01.702509 1455 tablet_copy_service.cc:140] P 31d5b9d4a2d2406a972f48086b62ceed: Received BeginTabletCopySession request for tablet 2bdda4e923dc4caaba534f1e8b5693e2 from peer b1330e34c41745c1bc55375408448018 ({username='slave'} at 127.0.0.1:52420)
I20240501 14:01:01.702621 1455 tablet_copy_service.cc:161] P 31d5b9d4a2d2406a972f48086b62ceed: Beginning new tablet copy session on tablet 2bdda4e923dc4caaba534f1e8b5693e2 from peer b1330e34c41745c1bc55375408448018 at {username='slave'} at 127.0.0.1:52420: session id = b1330e34c41745c1bc55375408448018-2bdda4e923dc4caaba534f1e8b5693e2
I20240501 14:01:01.705338 1455 tablet_copy_source_session.cc:205] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:01.706851 2279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bdda4e923dc4caaba534f1e8b5693e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.713421 2279 tablet_copy_client.cc:783] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:01.713732 2279 tablet_copy_client.cc:647] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:01.714320 2213 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=118 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:01.714712 2213 tablet_bootstrap.cc:492] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: Bootstrap complete.
I20240501 14:01:01.714967 2213 ts_tablet_manager.cc:1367] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: Time spent bootstrapping tablet: real 0.039s user 0.035s sys 0.006s
I20240501 14:01:01.715516 2283 tablet_copy_client.cc:1002] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:01.715674 2213 raft_consensus.cc:348] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:01.715871 2213 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8d30ece821d04a59962525029d0af320, State: Initialized, Role: LEARNER
I20240501 14:01:01.716065 2213 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:01.716598 2213 ts_tablet_manager.cc:1398] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:01.717288 1380 tablet_copy_service.cc:342] P c07691f747984bc7a2df956f4f212d8e: Request end of tablet copy session 8d30ece821d04a59962525029d0af320-6ee8e1d3ee3b429ebd15a32afe393e69 received from {username='slave'} at 127.0.0.1:45594
I20240501 14:01:01.717391 1380 tablet_copy_service.cc:433] P c07691f747984bc7a2df956f4f212d8e: ending tablet copy session 8d30ece821d04a59962525029d0af320-6ee8e1d3ee3b429ebd15a32afe393e69 on tablet 6ee8e1d3ee3b429ebd15a32afe393e69 with peer 8d30ece821d04a59962525029d0af320
I20240501 14:01:01.783128 2284 ts_tablet_manager.cc:902] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: Initiating tablet copy from peer b1330e34c41745c1bc55375408448018 (127.29.194.197:39411)
I20240501 14:01:01.784791 2284 tablet_copy_client.cc:308] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.197:39411
I20240501 14:01:01.785383 1849 tablet_copy_service.cc:140] P b1330e34c41745c1bc55375408448018: Received BeginTabletCopySession request for tablet ae2d33db85414876998a3ed550e44e41 from peer bf66d98ce8a34944be110a6fa9c75d09 ({username='slave'} at 127.0.0.1:57644)
I20240501 14:01:01.785573 1849 tablet_copy_service.cc:161] P b1330e34c41745c1bc55375408448018: Beginning new tablet copy session on tablet ae2d33db85414876998a3ed550e44e41 from peer bf66d98ce8a34944be110a6fa9c75d09 at {username='slave'} at 127.0.0.1:57644: session id = bf66d98ce8a34944be110a6fa9c75d09-ae2d33db85414876998a3ed550e44e41
I20240501 14:01:01.787631 1849 tablet_copy_source_session.cc:205] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:01.788733 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2d33db85414876998a3ed550e44e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.794603 2284 tablet_copy_client.cc:783] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:01.794850 2284 tablet_copy_client.cc:647] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:01.796329 2288 tablet_copy_client.cc:1002] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:01.808045 2289 ts_tablet_manager.cc:902] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: Initiating tablet copy from peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783)
I20240501 14:01:01.809676 2289 tablet_copy_client.cc:308] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.198:35783
I20240501 14:01:01.812670 2213 ts_tablet_manager.cc:902] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: Initiating tablet copy from peer 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:01:01.817299 1923 tablet_copy_service.cc:140] P 8d30ece821d04a59962525029d0af320: Received BeginTabletCopySession request for tablet 46ae855dc1a44159b5f3cca0657dc752 from peer bf66d98ce8a34944be110a6fa9c75d09 ({username='slave'} at 127.0.0.1:48030)
I20240501 14:01:01.817545 1923 tablet_copy_service.cc:161] P 8d30ece821d04a59962525029d0af320: Beginning new tablet copy session on tablet 46ae855dc1a44159b5f3cca0657dc752 from peer bf66d98ce8a34944be110a6fa9c75d09 at {username='slave'} at 127.0.0.1:48030: session id = bf66d98ce8a34944be110a6fa9c75d09-46ae855dc1a44159b5f3cca0657dc752
I20240501 14:01:01.817700 2213 tablet_copy_client.cc:308] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.195:38049
I20240501 14:01:01.818287 1529 tablet_copy_service.cc:140] P 1e896a0495ca458abad35d883aef4aef: Received BeginTabletCopySession request for tablet 7ca80506974245cb887559f7f884fb98 from peer 8d30ece821d04a59962525029d0af320 ({username='slave'} at 127.0.0.1:45064)
I20240501 14:01:01.818392 1529 tablet_copy_service.cc:161] P 1e896a0495ca458abad35d883aef4aef: Beginning new tablet copy session on tablet 7ca80506974245cb887559f7f884fb98 from peer 8d30ece821d04a59962525029d0af320 at {username='slave'} at 127.0.0.1:45064: session id = 8d30ece821d04a59962525029d0af320-7ca80506974245cb887559f7f884fb98
I20240501 14:01:01.819883 1923 tablet_copy_source_session.cc:205] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:01.820315 1529 tablet_copy_source_session.cc:205] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:01.820984 2289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46ae855dc1a44159b5f3cca0657dc752. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.821610 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ca80506974245cb887559f7f884fb98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.827678 2213 tablet_copy_client.cc:783] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:01.827706 2289 tablet_copy_client.cc:783] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:01.828017 2213 tablet_copy_client.cc:647] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:01.828061 2289 tablet_copy_client.cc:647] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:01.829656 2297 tablet_copy_client.cc:1002] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:01.829838 2293 tablet_copy_client.cc:1002] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:01.860255 2298 ts_tablet_manager.cc:902] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: Initiating tablet copy from peer 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:01:01.868865 2298 tablet_copy_client.cc:308] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.194:38379
I20240501 14:01:01.869437 1455 tablet_copy_service.cc:140] P 31d5b9d4a2d2406a972f48086b62ceed: Received BeginTabletCopySession request for tablet 78e839ad73564640b24fe9efc9f41f08 from peer 8d30ece821d04a59962525029d0af320 ({username='slave'} at 127.0.0.1:45488)
I20240501 14:01:01.869603 1455 tablet_copy_service.cc:161] P 31d5b9d4a2d2406a972f48086b62ceed: Beginning new tablet copy session on tablet 78e839ad73564640b24fe9efc9f41f08 from peer 8d30ece821d04a59962525029d0af320 at {username='slave'} at 127.0.0.1:45488: session id = 8d30ece821d04a59962525029d0af320-78e839ad73564640b24fe9efc9f41f08
I20240501 14:01:01.871924 1455 tablet_copy_source_session.cc:205] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:01.872910 2298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e839ad73564640b24fe9efc9f41f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:01.878115 2298 tablet_copy_client.cc:783] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:01.878314 2298 tablet_copy_client.cc:647] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:01.879693 2302 tablet_copy_client.cc:1002] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:01.939675 1912 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 2 LEARNER]: Advancing to term 4
I20240501 14:01:01.942286 1912 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 4 LEARNER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 14:01:01.942734 2151 consensus_queue.cc:1037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:01.947189 2306 raft_consensus.cc:1050] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
I20240501 14:01:01.947888 2306 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 31, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:01.950945 1444 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 14:01:01.951437 1370 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 14:01:01.951417 2307 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:01.951819 1912 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 4 LEARNER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 14:01:01.951946 2151 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:01.952997 2151 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:01.954489 2306 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:01.954977 1370 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:01.955789 1912 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:01.959493 1444 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:01.960911 1259 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 30 to 33, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:01.967559 1839 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:01.969081 1912 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240501 14:01:01.969568 2306 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:01.969895 1370 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240501 14:01:01.970364 2306 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:01.972182 2151 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:01.972834 1370 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:01.975253 1312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:01.976286 1267 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 reported cstate change: config changed from index 33 to 34, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New cstate: current_term: 4 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:01.976533 1912 raft_consensus.cc:2932] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 33 to 34, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:01.979908 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ee8e1d3ee3b429ebd15a32afe393e69 with delete_type TABLET_DATA_TOMBSTONED (TS 31d5b9d4a2d2406a972f48086b62ceed not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:55942
I20240501 14:01:01.980545 2312 tablet_replica.cc:330] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:01.980855 2312 raft_consensus.cc:2229] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:01.981348 2312 raft_consensus.cc:2258] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:01.982244 2312 ts_tablet_manager.cc:1875] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:01.987179 2312 ts_tablet_manager.cc:1888] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.33
I20240501 14:01:01.987390 2312 log.cc:1196] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root/wals/6ee8e1d3ee3b429ebd15a32afe393e69
I20240501 14:01:01.988070 1255 catalog_manager.cc:4830] TS 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379): tablet 6ee8e1d3ee3b429ebd15a32afe393e69 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:02.002671 2313 ts_tablet_manager.cc:902] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: Initiating tablet copy from peer bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675)
I20240501 14:01:02.004177 2313 tablet_copy_client.cc:308] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.196:41675
I20240501 14:01:02.004757 1774 tablet_copy_service.cc:140] P bf66d98ce8a34944be110a6fa9c75d09: Received BeginTabletCopySession request for tablet 355ae59edd1c4620904ba23275021582 from peer b1330e34c41745c1bc55375408448018 ({username='slave'} at 127.0.0.1:33434)
I20240501 14:01:02.004887 1774 tablet_copy_service.cc:161] P bf66d98ce8a34944be110a6fa9c75d09: Beginning new tablet copy session on tablet 355ae59edd1c4620904ba23275021582 from peer b1330e34c41745c1bc55375408448018 at {username='slave'} at 127.0.0.1:33434: session id = b1330e34c41745c1bc55375408448018-355ae59edd1c4620904ba23275021582
I20240501 14:01:02.006769 1774 tablet_copy_source_session.cc:205] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:02.007810 2313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 355ae59edd1c4620904ba23275021582. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:02.013263 2313 tablet_copy_client.cc:783] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:02.013466 2313 tablet_copy_client.cc:647] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:02.014824 2317 tablet_copy_client.cc:1002] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:02.020902 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:02.023962 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "407fed285f274f3c95342a21421b2c66"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:01:02.024166 1444 raft_consensus.cc:595] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Received request to transfer leadership to TS 1e896a0495ca458abad35d883aef4aef
I20240501 14:01:02.025141 1370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "8d30ece821d04a59962525029d0af320"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:01:02.025306 1370 raft_consensus.cc:595] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Received request to transfer leadership to TS 8d30ece821d04a59962525029d0af320
I20240501 14:01:02.025709 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240501 14:01:02.025939 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:02.226258 2319 raft_consensus.cc:979] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:01:02.226455 2319 raft_consensus.cc:1067] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Signalling peer 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:01:02.226980 1517 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "407fed285f274f3c95342a21421b2c66"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:56890
I20240501 14:01:02.227186 1517 raft_consensus.cc:482] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:02.227304 1517 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:02.229832 1517 raft_consensus.cc:504] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:02.230399 1517 leader_election.cc:288] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 3 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:01:02.230726 1439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:01:02.230953 1439 raft_consensus.cc:3032] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:01:02.231050 1439 raft_consensus.cc:729] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:02.231314 1439 consensus_queue.cc:260] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:02.231631 1439 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:02.230720 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:01:02.233891 1370 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:01:02.237406 1439 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 3.
I20240501 14:01:02.237421 1370 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 3.
I20240501 14:01:02.237742 1485 leader_election.cc:302] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:01:02.238363 2321 raft_consensus.cc:2781] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:01:02.238586 2321 raft_consensus.cc:686] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:01:02.238924 2321 consensus_queue.cc:237] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:02.242341 1267 catalog_manager.cc:5484] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 2 to 3, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 3 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:02.282171 2218 raft_consensus.cc:979] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 8d30ece821d04a59962525029d0af320 to start an election
I20240501 14:01:02.282370 2218 raft_consensus.cc:1067] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Signalling peer 8d30ece821d04a59962525029d0af320 to start an election
I20240501 14:01:02.282864 1913 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "916a3f31691044f783d655695bf0a369"
dest_uuid: "8d30ece821d04a59962525029d0af320"
from {username='slave'} at 127.0.0.1:38324
I20240501 14:01:02.283058 1913 raft_consensus.cc:482] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:02.283185 1913 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:02.286212 1913 raft_consensus.cc:504] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:02.287340 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "8d30ece821d04a59962525029d0af320" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:01:02.287479 1913 leader_election.cc:288] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [CANDIDATE]: Term 5 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:01:02.287542 1370 raft_consensus.cc:3032] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:02.287542 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916a3f31691044f783d655695bf0a369" candidate_uuid: "8d30ece821d04a59962525029d0af320" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:01:02.287668 1370 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:02.287791 1517 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:02.287962 1370 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:02.288342 1370 raft_consensus.cc:3037] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:02.290786 1517 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d30ece821d04a59962525029d0af320 in term 5.
I20240501 14:01:02.291095 1370 raft_consensus.cc:2454] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d30ece821d04a59962525029d0af320 in term 5.
I20240501 14:01:02.291175 1877 leader_election.cc:302] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 8d30ece821d04a59962525029d0af320; no voters:
I20240501 14:01:02.291465 2124 raft_consensus.cc:2781] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:02.291604 2124 raft_consensus.cc:686] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 5 LEADER]: Becoming Leader. State: Replica: 8d30ece821d04a59962525029d0af320, State: Running, Role: LEADER
I20240501 14:01:02.291852 2124 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:02.294620 1267 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 reported cstate change: term changed from 4 to 5, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 8d30ece821d04a59962525029d0af320 (127.29.194.198). New cstate: current_term: 5 leader_uuid: "8d30ece821d04a59962525029d0af320" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:02.687837 1370 raft_consensus.cc:1261] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:01:02.688361 2321 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:01:02.692795 1517 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240501 14:01:02.693325 2124 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:02.702421 1444 raft_consensus.cc:1261] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 14:01:02.703066 2322 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 14:01:02.704452 1370 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240501 14:01:02.705389 2124 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:03.715831 2279 tablet_copy_client.cc:523] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:03.718713 2279 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: Bootstrap starting.
I20240501 14:01:03.770224 2279 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=113 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:03.770570 2279 tablet_bootstrap.cc:492] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: Bootstrap complete.
I20240501 14:01:03.770808 2279 ts_tablet_manager.cc:1367] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: Time spent bootstrapping tablet: real 0.052s user 0.049s sys 0.003s
I20240501 14:01:03.771399 2279 raft_consensus.cc:348] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:03.771576 2279 raft_consensus.cc:729] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: b1330e34c41745c1bc55375408448018, State: Initialized, Role: LEARNER
I20240501 14:01:03.771787 2279 consensus_queue.cc:260] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:03.772291 2279 ts_tablet_manager.cc:1398] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:01:03.772890 1455 tablet_copy_service.cc:342] P 31d5b9d4a2d2406a972f48086b62ceed: Request end of tablet copy session b1330e34c41745c1bc55375408448018-2bdda4e923dc4caaba534f1e8b5693e2 received from {username='slave'} at 127.0.0.1:52420
I20240501 14:01:03.772979 1455 tablet_copy_service.cc:433] P 31d5b9d4a2d2406a972f48086b62ceed: ending tablet copy session b1330e34c41745c1bc55375408448018-2bdda4e923dc4caaba534f1e8b5693e2 on tablet 2bdda4e923dc4caaba534f1e8b5693e2 with peer b1330e34c41745c1bc55375408448018
I20240501 14:01:03.796675 2284 tablet_copy_client.cc:523] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:03.799906 2284 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap starting.
I20240501 14:01:03.830089 2213 tablet_copy_client.cc:523] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:03.830178 2289 tablet_copy_client.cc:523] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:03.833134 2213 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: Bootstrap starting.
I20240501 14:01:03.833184 2289 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap starting.
I20240501 14:01:03.852285 2284 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=123 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:03.852756 2284 tablet_bootstrap.cc:492] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap complete.
I20240501 14:01:03.853070 2284 ts_tablet_manager.cc:1367] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: Time spent bootstrapping tablet: real 0.053s user 0.038s sys 0.004s
I20240501 14:01:03.853878 2284 raft_consensus.cc:348] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:01:03.854231 2284 raft_consensus.cc:729] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: bf66d98ce8a34944be110a6fa9c75d09, State: Initialized, Role: LEARNER
I20240501 14:01:03.854521 2284 consensus_queue.cc:260] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 3.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:01:03.855799 2284 ts_tablet_manager.cc:1398] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:03.857021 1849 tablet_copy_service.cc:342] P b1330e34c41745c1bc55375408448018: Request end of tablet copy session bf66d98ce8a34944be110a6fa9c75d09-ae2d33db85414876998a3ed550e44e41 received from {username='slave'} at 127.0.0.1:57644
I20240501 14:01:03.857117 1849 tablet_copy_service.cc:433] P b1330e34c41745c1bc55375408448018: ending tablet copy session bf66d98ce8a34944be110a6fa9c75d09-ae2d33db85414876998a3ed550e44e41 on tablet ae2d33db85414876998a3ed550e44e41 with peer bf66d98ce8a34944be110a6fa9c75d09
I20240501 14:01:03.877360 2289 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:03.877816 2289 tablet_bootstrap.cc:492] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap complete.
I20240501 14:01:03.878093 2289 ts_tablet_manager.cc:1367] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: Time spent bootstrapping tablet: real 0.045s user 0.036s sys 0.006s
I20240501 14:01:03.878927 2289 raft_consensus.cc:348] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:01:03.879231 2289 raft_consensus.cc:729] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: bf66d98ce8a34944be110a6fa9c75d09, State: Initialized, Role: LEARNER
I20240501 14:01:03.879469 2289 consensus_queue.cc:260] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:01:03.880018 2289 ts_tablet_manager.cc:1398] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:03.880137 2298 tablet_copy_client.cc:523] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:03.880842 1923 tablet_copy_service.cc:342] P 8d30ece821d04a59962525029d0af320: Request end of tablet copy session bf66d98ce8a34944be110a6fa9c75d09-46ae855dc1a44159b5f3cca0657dc752 received from {username='slave'} at 127.0.0.1:48030
I20240501 14:01:03.880956 1923 tablet_copy_service.cc:433] P 8d30ece821d04a59962525029d0af320: ending tablet copy session bf66d98ce8a34944be110a6fa9c75d09-46ae855dc1a44159b5f3cca0657dc752 on tablet 46ae855dc1a44159b5f3cca0657dc752 with peer bf66d98ce8a34944be110a6fa9c75d09
I20240501 14:01:03.883531 2298 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: Bootstrap starting.
I20240501 14:01:03.893026 2213 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:03.893421 2213 tablet_bootstrap.cc:492] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: Bootstrap complete.
I20240501 14:01:03.893663 2213 ts_tablet_manager.cc:1367] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: Time spent bootstrapping tablet: real 0.061s user 0.039s sys 0.004s
I20240501 14:01:03.894392 2213 raft_consensus.cc:348] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:03.894661 2213 raft_consensus.cc:729] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d30ece821d04a59962525029d0af320, State: Initialized, Role: LEARNER
I20240501 14:01:03.894893 2213 consensus_queue.cc:260] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:03.895445 2213 ts_tablet_manager.cc:1398] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:01:03.896102 1529 tablet_copy_service.cc:342] P 1e896a0495ca458abad35d883aef4aef: Request end of tablet copy session 8d30ece821d04a59962525029d0af320-7ca80506974245cb887559f7f884fb98 received from {username='slave'} at 127.0.0.1:45064
I20240501 14:01:03.896206 1529 tablet_copy_service.cc:433] P 1e896a0495ca458abad35d883aef4aef: ending tablet copy session 8d30ece821d04a59962525029d0af320-7ca80506974245cb887559f7f884fb98 on tablet 7ca80506974245cb887559f7f884fb98 with peer 8d30ece821d04a59962525029d0af320
I20240501 14:01:03.927788 2298 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:03.928207 2298 tablet_bootstrap.cc:492] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: Bootstrap complete.
I20240501 14:01:03.928465 2298 ts_tablet_manager.cc:1367] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: Time spent bootstrapping tablet: real 0.045s user 0.038s sys 0.004s
I20240501 14:01:03.929208 2298 raft_consensus.cc:348] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:03.929404 2298 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 8d30ece821d04a59962525029d0af320, State: Initialized, Role: LEARNER
I20240501 14:01:03.929597 2298 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 5.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: NON_VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: true } }
I20240501 14:01:03.930176 2298 ts_tablet_manager.cc:1398] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:03.930944 1455 tablet_copy_service.cc:342] P 31d5b9d4a2d2406a972f48086b62ceed: Request end of tablet copy session 8d30ece821d04a59962525029d0af320-78e839ad73564640b24fe9efc9f41f08 received from {username='slave'} at 127.0.0.1:45488
I20240501 14:01:03.931073 1455 tablet_copy_service.cc:433] P 31d5b9d4a2d2406a972f48086b62ceed: ending tablet copy session 8d30ece821d04a59962525029d0af320-78e839ad73564640b24fe9efc9f41f08 on tablet 78e839ad73564640b24fe9efc9f41f08 with peer 8d30ece821d04a59962525029d0af320
I20240501 14:01:03.950949 1913 raft_consensus.cc:1203] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.35->[5.36-5.36] Dedup: 5.36->[]
I20240501 14:01:04.015205 2313 tablet_copy_client.cc:523] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:04.018548 2313 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: Bootstrap starting.
I20240501 14:01:04.026299 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:04.030092 1517 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "407fed285f274f3c95342a21421b2c66"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:01:04.030267 1517 raft_consensus.cc:595] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:01:04.030786 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:04.030997 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:04.056607 2313 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:04.057112 2313 tablet_bootstrap.cc:492] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: Bootstrap complete.
I20240501 14:01:04.057408 2313 ts_tablet_manager.cc:1367] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: Time spent bootstrapping tablet: real 0.039s user 0.039s sys 0.000s
I20240501 14:01:04.058071 2313 raft_consensus.cc:348] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:04.058243 2313 raft_consensus.cc:729] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b1330e34c41745c1bc55375408448018, State: Initialized, Role: LEARNER
I20240501 14:01:04.058413 2313 consensus_queue.cc:260] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:04.058928 2313 ts_tablet_manager.cc:1398] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:01:04.059595 1774 tablet_copy_service.cc:342] P bf66d98ce8a34944be110a6fa9c75d09: Request end of tablet copy session b1330e34c41745c1bc55375408448018-355ae59edd1c4620904ba23275021582 received from {username='slave'} at 127.0.0.1:33434
I20240501 14:01:04.059703 1774 tablet_copy_service.cc:433] P bf66d98ce8a34944be110a6fa9c75d09: ending tablet copy session b1330e34c41745c1bc55375408448018-355ae59edd1c4620904ba23275021582 on tablet 355ae59edd1c4620904ba23275021582 with peer b1330e34c41745c1bc55375408448018
I20240501 14:01:04.102403 1764 raft_consensus.cc:1203] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.33->[3.34-3.34] Dedup: 3.34->[]
I20240501 14:01:04.142321 1839 raft_consensus.cc:1203] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.30->[4.31-4.31] Dedup: 4.31->[]
I20240501 14:01:04.180599 2338 raft_consensus.cc:979] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:04.180841 2320 raft_consensus.cc:1067] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:04.181388 1370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "407fed285f274f3c95342a21421b2c66"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58886
I20240501 14:01:04.181556 1370 raft_consensus.cc:482] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:04.181651 1370 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:04.183833 1370 raft_consensus.cc:504] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:04.184921 1370 leader_election.cc:288] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 4 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:01:04.184921 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:01:04.184988 1517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:01:04.185171 1444 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:04.185189 1517 raft_consensus.cc:3032] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:04.185273 1517 raft_consensus.cc:729] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:01:04.185477 1517 consensus_queue.cc:260] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:04.185715 1517 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:04.187059 1444 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 4.
I20240501 14:01:04.187381 1336 leader_election.cc:302] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:01:04.187536 1517 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 4.
I20240501 14:01:04.187624 2218 raft_consensus.cc:2781] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:04.188196 2218 raft_consensus.cc:686] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:04.188413 2218 consensus_queue.cc:237] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:04.190878 1267 catalog_manager.cc:5484] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 3 to 4, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 4 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:04.317122 1839 raft_consensus.cc:1203] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20240501 14:01:04.341519 1913 raft_consensus.cc:1203] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30] Dedup: 1.30->[]
I20240501 14:01:04.353832 2318 raft_consensus.cc:1050] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
I20240501 14:01:04.354661 2318 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 5.36, Last appended by leader: 34, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:04.356936 1839 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240501 14:01:04.356936 1913 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 5 LEARNER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240501 14:01:04.357626 1517 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240501 14:01:04.357848 2318 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240501 14:01:04.358121 2354 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240501 14:01:04.358505 2355 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240501 14:01:04.362499 1913 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.362617 1839 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.361990 2355 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Committing config change with OpId 5.37: config changed from index 36 to 37, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.368037 1267 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 reported cstate change: config changed from index 36 to 37, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.370676 1517 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.376250 1443 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 5.37, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:04.377790 1913 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 5 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20240501 14:01:04.378036 1839 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 5 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20240501 14:01:04.378515 2352 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240501 14:01:04.379163 2354 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240501 14:01:04.380417 1763 raft_consensus.cc:1203] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.32->[2.33-2.33] Dedup: 2.33->[]
I20240501 14:01:04.382097 2318 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.382388 1838 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.385823 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 78e839ad73564640b24fe9efc9f41f08 with cas_config_opid_index 37: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:04.385552 1913 raft_consensus.cc:2932] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.386420 1268 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 37 to 38, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New cstate: current_term: 5 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:04.389916 1508 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e839ad73564640b24fe9efc9f41f08 with delete_type TABLET_DATA_TOMBSTONED (TS 1e896a0495ca458abad35d883aef4aef not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:56882
I20240501 14:01:04.390483 2359 tablet_replica.cc:330] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:04.390813 2359 raft_consensus.cc:2229] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.391115 2359 raft_consensus.cc:2258] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.391995 2359 ts_tablet_manager.cc:1875] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.396939 2359 ts_tablet_manager.cc:1888] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.37
I20240501 14:01:04.397025 2359 log.cc:1196] T 78e839ad73564640b24fe9efc9f41f08 P 1e896a0495ca458abad35d883aef4aef: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root/wals/78e839ad73564640b24fe9efc9f41f08
I20240501 14:01:04.397835 1253 catalog_manager.cc:4830] TS 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): tablet 78e839ad73564640b24fe9efc9f41f08 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:04.526103 2339 raft_consensus.cc:1050] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: attempting to promote NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 to VOTER
I20240501 14:01:04.526955 2339 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 32, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:04.530623 1517 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20240501 14:01:04.530644 1763 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 LEARNER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20240501 14:01:04.531030 1370 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20240501 14:01:04.531200 2362 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:04.531756 2360 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:04.532045 2339 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:04.535552 2362 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 LEADER]: Committing config change with OpId 3.35: config changed from index 34 to 35, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.536705 1517 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.540803 1370 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.541080 1268 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index 34 to 35, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.545539 1763 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.548815 1839 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 3.35, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:04.550273 1763 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20240501 14:01:04.550707 1517 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 3 index: 35. Preceding OpId from leader: term: 3 index: 36. (index mismatch)
I20240501 14:01:04.550781 2363 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 14:01:04.551420 2360 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 14:01:04.553265 2363 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 LEADER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.554680 1517 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.556867 1763 raft_consensus.cc:2932] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Committing config change with OpId 3.36: config changed from index 35 to 36, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.557314 1312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae2d33db85414876998a3ed550e44e41 with cas_config_opid_index 35: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:04.559108 1268 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 reported cstate change: config changed from index 35 to 36, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New cstate: current_term: 3 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:04.562726 1360 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2d33db85414876998a3ed550e44e41 with delete_type TABLET_DATA_TOMBSTONED (TS c07691f747984bc7a2df956f4f212d8e not found in new config with opid_index 36) from {username='slave'} at 127.0.0.1:58866
I20240501 14:01:04.565261 2368 tablet_replica.cc:330] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:04.565472 2368 raft_consensus.cc:2229] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.565634 2368 raft_consensus.cc:2258] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.566452 2368 ts_tablet_manager.cc:1875] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.571904 2368 ts_tablet_manager.cc:1888] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.35
I20240501 14:01:04.572000 2368 log.cc:1196] T ae2d33db85414876998a3ed550e44e41 P c07691f747984bc7a2df956f4f212d8e: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root/wals/ae2d33db85414876998a3ed550e44e41
I20240501 14:01:04.572726 1252 catalog_manager.cc:4830] TS c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): tablet ae2d33db85414876998a3ed550e44e41 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:04.637950 2352 raft_consensus.cc:1050] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: attempting to promote NON_VOTER b1330e34c41745c1bc55375408448018 to VOTER
I20240501 14:01:04.638686 2352 consensus_queue.cc:237] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 29, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:04.641155 1839 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 LEARNER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:01:04.641505 1370 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:01:04.641786 2352 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:04.642043 2355 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:04.643033 1517 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240501 14:01:04.643430 1444 raft_consensus.cc:1261] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 14:01:04.644023 2218 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:01:04.645040 2355 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:04.647552 1370 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.651541 2355 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.658917 1839 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.659152 1259 catalog_manager.cc:5484] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 31 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.662498 1519 raft_consensus.cc:1261] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 14:01:04.662957 2345 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:01:04.662712 1514 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.665114 1443 consensus_queue.cc:237] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:04.666594 1839 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 14:01:04.667099 1370 raft_consensus.cc:1261] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240501 14:01:04.667198 2352 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:04.667766 2352 consensus_queue.cc:1035] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:04.669459 2351 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.672232 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2bdda4e923dc4caaba534f1e8b5693e2 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:04.673413 1370 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.673642 1261 catalog_manager.cc:5484] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 32 to 33, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New cstate: current_term: 4 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:04.676635 1839 raft_consensus.cc:2932] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.676898 1508 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bdda4e923dc4caaba534f1e8b5693e2 with delete_type TABLET_DATA_TOMBSTONED (TS 1e896a0495ca458abad35d883aef4aef not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:56882
I20240501 14:01:04.677183 2359 tablet_replica.cc:330] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:04.677342 2359 raft_consensus.cc:2229] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.677548 2359 raft_consensus.cc:2258] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.678433 2359 ts_tablet_manager.cc:1875] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.685765 2359 ts_tablet_manager.cc:1888] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.32
I20240501 14:01:04.685873 2359 log.cc:1196] T 2bdda4e923dc4caaba534f1e8b5693e2 P 1e896a0495ca458abad35d883aef4aef: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root/wals/2bdda4e923dc4caaba534f1e8b5693e2
I20240501 14:01:04.686962 1253 catalog_manager.cc:4830] TS 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): tablet 2bdda4e923dc4caaba534f1e8b5693e2 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:04.748652 2378 raft_consensus.cc:1050] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: attempting to promote NON_VOTER 8d30ece821d04a59962525029d0af320 to VOTER
I20240501 14:01:04.749449 2378 consensus_queue.cc:237] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:04.751892 1763 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 14:01:04.751926 1443 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 14:01:04.752192 1912 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 LEARNER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 14:01:04.752516 2381 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:04.752975 2379 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:04.753240 2378 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:04.757903 2378 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.758587 1762 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.762301 1443 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.763357 1259 catalog_manager.cc:5484] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index 30 to 31, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:04.772886 1912 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 8d30ece821d04a59962525029d0af320 (127.29.194.198) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.775681 1519 consensus_queue.cc:237] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:04.777606 1913 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240501 14:01:04.778132 2378 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:04.778618 1762 raft_consensus.cc:1261] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240501 14:01:04.779084 2378 consensus_queue.cc:1035] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:04.780980 2379 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.781518 1913 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.781539 1763 raft_consensus.cc:2932] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } }
I20240501 14:01:04.784548 1253 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7ca80506974245cb887559f7f884fb98 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:04.786216 1261 catalog_manager.cc:5484] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef reported cstate change: config changed from index 31 to 32, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New cstate: current_term: 1 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:04.789582 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ca80506974245cb887559f7f884fb98 with delete_type TABLET_DATA_TOMBSTONED (TS 31d5b9d4a2d2406a972f48086b62ceed not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:55942
I20240501 14:01:04.790222 2385 tablet_replica.cc:330] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:04.790470 2385 raft_consensus.cc:2229] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.790719 2385 raft_consensus.cc:2258] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.791522 2385 ts_tablet_manager.cc:1875] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.797195 2385 ts_tablet_manager.cc:1888] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.31
I20240501 14:01:04.797315 2385 log.cc:1196] T 7ca80506974245cb887559f7f884fb98 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root/wals/7ca80506974245cb887559f7f884fb98
I20240501 14:01:04.798023 1255 catalog_manager.cc:4830] TS 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379): tablet 7ca80506974245cb887559f7f884fb98 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:04.818675 2328 raft_consensus.cc:1050] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: attempting to promote NON_VOTER b1330e34c41745c1bc55375408448018 to VOTER
I20240501 14:01:04.819501 2328 consensus_queue.cc:237] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:04.821455 1839 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:01:04.821795 1443 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:01:04.821894 2386 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:04.822665 2386 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:04.825901 2386 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.826951 1443 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.830852 1268 catalog_manager.cc:5484] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "bf66d98ce8a34944be110a6fa9c75d09" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.834443 1370 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:01:04.834951 2328 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:04.836040 1839 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.837144 1370 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.839329 1763 consensus_queue.cc:237] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:04.841475 1839 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 14:01:04.841728 1443 raft_consensus.cc:1261] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 14:01:04.842198 2386 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:04.842553 2386 consensus_queue.cc:1035] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:04.844579 2328 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.845255 1443 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.847976 1312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 355ae59edd1c4620904ba23275021582 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:04.849484 1259 catalog_manager.cc:5484] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 33 to 34, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New cstate: current_term: 2 leader_uuid: "bf66d98ce8a34944be110a6fa9c75d09" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.850484 1839 raft_consensus.cc:2932] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:04.852631 1360 tablet_service.cc:1514] Processing DeleteTablet for tablet 355ae59edd1c4620904ba23275021582 with delete_type TABLET_DATA_TOMBSTONED (TS c07691f747984bc7a2df956f4f212d8e not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:58866
I20240501 14:01:04.852916 2368 tablet_replica.cc:330] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:04.853086 2368 raft_consensus.cc:2229] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.853257 2368 raft_consensus.cc:2258] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.855275 2368 ts_tablet_manager.cc:1875] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.860810 2368 ts_tablet_manager.cc:1888] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240501 14:01:04.860951 2368 log.cc:1196] T 355ae59edd1c4620904ba23275021582 P c07691f747984bc7a2df956f4f212d8e: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root/wals/355ae59edd1c4620904ba23275021582
I20240501 14:01:04.861565 1252 catalog_manager.cc:4830] TS c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): tablet 355ae59edd1c4620904ba23275021582 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:04.912282 2394 raft_consensus.cc:1050] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: attempting to promote NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 to VOTER
I20240501 14:01:04.913121 2394 consensus_queue.cc:237] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 31, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:04.915320 1763 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEARNER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 14:01:04.915290 1514 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 14:01:04.915283 1443 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 14:01:04.915931 2397 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:04.916399 2305 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:04.916657 2394 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:04.919773 2305 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.922739 1443 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.925904 1514 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.928975 1763 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.929265 1268 catalog_manager.cc:5484] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: config changed from index 33 to 34, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8d30ece821d04a59962525029d0af320" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.936414 1913 consensus_queue.cc:237] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:04.937788 1763 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240501 14:01:04.938158 1514 raft_consensus.cc:1261] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20240501 14:01:04.938246 2394 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:04.938535 2397 consensus_queue.cc:1035] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:04.940466 2305 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.940992 1763 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.941273 1514 raft_consensus.cc:2932] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:04.943367 1310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46ae855dc1a44159b5f3cca0657dc752 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:04.944285 1261 catalog_manager.cc:5484] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 reported cstate change: config changed from index 34 to 35, VOTER 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) evicted. New cstate: current_term: 2 leader_uuid: "8d30ece821d04a59962525029d0af320" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:04.948074 1427 tablet_service.cc:1514] Processing DeleteTablet for tablet 46ae855dc1a44159b5f3cca0657dc752 with delete_type TABLET_DATA_TOMBSTONED (TS 31d5b9d4a2d2406a972f48086b62ceed not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:55942
I20240501 14:01:04.948333 2385 tablet_replica.cc:330] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:04.948450 2385 raft_consensus.cc:2229] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:04.948622 2385 raft_consensus.cc:2258] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:04.949441 2385 ts_tablet_manager.cc:1875] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:04.954072 2385 ts_tablet_manager.cc:1888] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.34
I20240501 14:01:04.954166 2385 log.cc:1196] T 46ae855dc1a44159b5f3cca0657dc752 P 31d5b9d4a2d2406a972f48086b62ceed: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-1-root/wals/46ae855dc1a44159b5f3cca0657dc752
I20240501 14:01:04.954759 1255 catalog_manager.cc:4830] TS 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379): tablet 46ae855dc1a44159b5f3cca0657dc752 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:05.949318 1912 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
W20240501 14:01:05.951829 1877 consensus_peers.cc:485] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 -> Peer bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196:41675): Couldn't send request to peer bf66d98ce8a34944be110a6fa9c75d09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 916a3f31691044f783d655695bf0a369. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:05.952469 1370 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240501 14:01:05.952733 1514 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 5 index: 36. (index mismatch)
I20240501 14:01:05.952991 2394 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 14:01:05.953274 2403 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 14:01:05.956144 2394 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 5 LEADER]: Committing config change with OpId 5.36: config changed from index 34 to 36, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:05.956620 1519 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 34 to 36, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:05.957527 1370 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Committing config change with OpId 5.36: config changed from index 34 to 36, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } } }
I20240501 14:01:05.961681 1259 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 reported cstate change: config changed from index 34 to 36, NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) added. New cstate: current_term: 5 leader_uuid: "8d30ece821d04a59962525029d0af320" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:05.963299 1370 consensus_queue.cc:237] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:05.966064 1912 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
W20240501 14:01:05.966858 1334 consensus_peers.cc:485] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e -> Peer b1330e34c41745c1bc55375408448018 (127.29.194.197:39411): Couldn't send request to peer b1330e34c41745c1bc55375408448018. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c36a1a8c09d415d8c1d1860c0d8b622. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:05.967172 2345 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:05.969143 1519 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 14:01:05.969879 2412 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:05.970306 2414 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:05.970913 1912 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:05.977020 1519 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:05.977254 1261 catalog_manager.cc:5484] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 reported cstate change: config changed from index 31 to 32, NON_VOTER b1330e34c41745c1bc55375408448018 (127.29.194.197) added. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } } }
I20240501 14:01:06.031282 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:06.034363 1370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "407fed285f274f3c95342a21421b2c66"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:01:06.034636 1370 raft_consensus.cc:595] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Received request to transfer leadership to TS 1e896a0495ca458abad35d883aef4aef
I20240501 14:01:06.035094 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:06.035315 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:06.332605 2416 ts_tablet_manager.cc:902] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: Initiating tablet copy from peer 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783)
I20240501 14:01:06.333935 2416 tablet_copy_client.cc:308] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.198:35783
I20240501 14:01:06.334570 1923 tablet_copy_service.cc:140] P 8d30ece821d04a59962525029d0af320: Received BeginTabletCopySession request for tablet 916a3f31691044f783d655695bf0a369 from peer bf66d98ce8a34944be110a6fa9c75d09 ({username='slave'} at 127.0.0.1:48030)
I20240501 14:01:06.334690 1923 tablet_copy_service.cc:161] P 8d30ece821d04a59962525029d0af320: Beginning new tablet copy session on tablet 916a3f31691044f783d655695bf0a369 from peer bf66d98ce8a34944be110a6fa9c75d09 at {username='slave'} at 127.0.0.1:48030: session id = bf66d98ce8a34944be110a6fa9c75d09-916a3f31691044f783d655695bf0a369
I20240501 14:01:06.337231 1923 tablet_copy_source_session.cc:205] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:06.338325 2416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916a3f31691044f783d655695bf0a369. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.343359 2416 tablet_copy_client.cc:783] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:06.343554 2416 tablet_copy_client.cc:647] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:06.345171 2420 tablet_copy_client.cc:1002] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:06.399107 2414 raft_consensus.cc:979] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:01:06.399264 2414 raft_consensus.cc:1067] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Signalling peer 1e896a0495ca458abad35d883aef4aef to start an election
I20240501 14:01:06.399823 1519 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "407fed285f274f3c95342a21421b2c66"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
from {username='slave'} at 127.0.0.1:56892
I20240501 14:01:06.400051 1519 raft_consensus.cc:482] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:06.400159 1519 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:06.402431 1519 raft_consensus.cc:504] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:06.403189 1519 leader_election.cc:288] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 5 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379)
I20240501 14:01:06.403574 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:01:06.403752 1444 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:06.403728 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407fed285f274f3c95342a21421b2c66" candidate_uuid: "1e896a0495ca458abad35d883aef4aef" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:01:06.404991 1370 raft_consensus.cc:3032] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:06.405089 1370 raft_consensus.cc:729] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:06.405324 1370 consensus_queue.cc:260] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:06.405715 1370 raft_consensus.cc:3037] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:06.406023 1444 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 5.
I20240501 14:01:06.406410 1485 leader_election.cc:302] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:01:06.406663 2381 raft_consensus.cc:2781] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:06.407061 2381 raft_consensus.cc:686] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 5 LEADER]: Becoming Leader. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:01:06.407253 2381 consensus_queue.cc:237] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:06.407903 1370 raft_consensus.cc:2454] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e896a0495ca458abad35d883aef4aef in term 5.
I20240501 14:01:06.409658 1261 catalog_manager.cc:5484] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef reported cstate change: term changed from 4 to 5, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 1e896a0495ca458abad35d883aef4aef (127.29.194.195). New cstate: current_term: 5 leader_uuid: "1e896a0495ca458abad35d883aef4aef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } }
I20240501 14:01:06.425885 2423 ts_tablet_manager.cc:902] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: Initiating tablet copy from peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821)
I20240501 14:01:06.427141 2423 tablet_copy_client.cc:308] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.193:46821
I20240501 14:01:06.427688 1380 tablet_copy_service.cc:140] P c07691f747984bc7a2df956f4f212d8e: Received BeginTabletCopySession request for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 from peer b1330e34c41745c1bc55375408448018 ({username='slave'} at 127.0.0.1:45572)
I20240501 14:01:06.427812 1380 tablet_copy_service.cc:161] P c07691f747984bc7a2df956f4f212d8e: Beginning new tablet copy session on tablet 8c36a1a8c09d415d8c1d1860c0d8b622 from peer b1330e34c41745c1bc55375408448018 at {username='slave'} at 127.0.0.1:45572: session id = b1330e34c41745c1bc55375408448018-8c36a1a8c09d415d8c1d1860c0d8b622
I20240501 14:01:06.430053 1380 tablet_copy_source_session.cc:205] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:06.431103 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c36a1a8c09d415d8c1d1860c0d8b622. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:06.436424 2423 tablet_copy_client.cc:783] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:06.436635 2423 tablet_copy_client.cc:647] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:06.438055 2427 tablet_copy_client.cc:1002] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: tablet copy: Injecting latency into file download: 2000
I20240501 14:01:06.831197 1444 raft_consensus.cc:1261] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240501 14:01:06.831727 2381 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:06.836010 1367 raft_consensus.cc:1261] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Refusing update from remote peer 1e896a0495ca458abad35d883aef4aef: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240501 14:01:06.836549 2421 consensus_queue.cc:1035] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:08.035683 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:08.038971 1519 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "1e896a0495ca458abad35d883aef4aef"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45054
I20240501 14:01:08.039193 1519 raft_consensus.cc:595] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:01:08.039587 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:08.039803 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:08.345553 2416 tablet_copy_client.cc:523] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:08.349040 2416 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap starting.
I20240501 14:01:08.391364 2416 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:08.391824 2416 tablet_bootstrap.cc:492] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: Bootstrap complete.
I20240501 14:01:08.392110 2416 ts_tablet_manager.cc:1367] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: Time spent bootstrapping tablet: real 0.043s user 0.036s sys 0.004s
I20240501 14:01:08.392832 2416 raft_consensus.cc:348] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:01:08.393049 2416 raft_consensus.cc:729] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: bf66d98ce8a34944be110a6fa9c75d09, State: Initialized, Role: LEARNER
I20240501 14:01:08.393253 2416 consensus_queue.cc:260] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 5.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: true } }
I20240501 14:01:08.393875 2416 ts_tablet_manager.cc:1398] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240501 14:01:08.394495 1923 tablet_copy_service.cc:342] P 8d30ece821d04a59962525029d0af320: Request end of tablet copy session bf66d98ce8a34944be110a6fa9c75d09-916a3f31691044f783d655695bf0a369 received from {username='slave'} at 127.0.0.1:48030
I20240501 14:01:08.394606 1923 tablet_copy_service.cc:433] P 8d30ece821d04a59962525029d0af320: ending tablet copy session bf66d98ce8a34944be110a6fa9c75d09-916a3f31691044f783d655695bf0a369 on tablet 916a3f31691044f783d655695bf0a369 with peer bf66d98ce8a34944be110a6fa9c75d09
I20240501 14:01:08.438385 2423 tablet_copy_client.cc:523] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:08.441269 2423 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: Bootstrap starting.
I20240501 14:01:08.455317 2440 raft_consensus.cc:979] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:08.455533 2440 raft_consensus.cc:1067] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:08.456162 1367 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58886
I20240501 14:01:08.456386 1367 raft_consensus.cc:482] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:08.456475 1367 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:08.459204 1367 raft_consensus.cc:504] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:08.460192 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:01:08.460527 1444 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:08.460585 1367 leader_election.cc:288] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 4 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:01:08.463109 1444 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 4.
I20240501 14:01:08.463426 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:01:08.463601 1519 raft_consensus.cc:3032] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:08.463615 1336 leader_election.cc:302] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:01:08.463682 1519 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1e896a0495ca458abad35d883aef4aef, State: Running, Role: LEADER
I20240501 14:01:08.463956 1519 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:08.464303 1519 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:08.466343 1519 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 4.
I20240501 14:01:08.466691 2345 raft_consensus.cc:2781] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:08.467221 2345 raft_consensus.cc:686] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:08.467470 2345 consensus_queue.cc:237] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:08.470180 1268 catalog_manager.cc:5484] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 3 to 4, leader changed from 1e896a0495ca458abad35d883aef4aef (127.29.194.195) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 4 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:08.481264 1763 raft_consensus.cc:1203] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.35->[5.36-5.36] Dedup: 5.36->[]
I20240501 14:01:08.498775 2423 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:08.499176 2423 tablet_bootstrap.cc:492] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: Bootstrap complete.
I20240501 14:01:08.499430 2423 ts_tablet_manager.cc:1367] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: Time spent bootstrapping tablet: real 0.058s user 0.054s sys 0.001s
I20240501 14:01:08.500109 2423 raft_consensus.cc:348] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:08.500319 2423 raft_consensus.cc:729] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b1330e34c41745c1bc55375408448018, State: Initialized, Role: LEARNER
I20240501 14:01:08.500515 2423 consensus_queue.cc:260] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: NON_VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: true } }
I20240501 14:01:08.501057 2423 ts_tablet_manager.cc:1398] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240501 14:01:08.501672 1380 tablet_copy_service.cc:342] P c07691f747984bc7a2df956f4f212d8e: Request end of tablet copy session b1330e34c41745c1bc55375408448018-8c36a1a8c09d415d8c1d1860c0d8b622 received from {username='slave'} at 127.0.0.1:45572
I20240501 14:01:08.501775 1380 tablet_copy_service.cc:433] P c07691f747984bc7a2df956f4f212d8e: ending tablet copy session b1330e34c41745c1bc55375408448018-8c36a1a8c09d415d8c1d1860c0d8b622 on tablet 8c36a1a8c09d415d8c1d1860c0d8b622 with peer b1330e34c41745c1bc55375408448018
I20240501 14:01:08.778532 1839 raft_consensus.cc:1203] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20240501 14:01:08.885519 1444 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 14:01:08.886023 2345 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:01:08.901932 1519 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240501 14:01:08.902933 2441 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 14:01:08.982911 2454 raft_consensus.cc:1050] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: attempting to promote NON_VOTER bf66d98ce8a34944be110a6fa9c75d09 to VOTER
I20240501 14:01:08.983824 2454 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 5.36, Last appended by leader: 34, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:08.985930 1763 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 LEARNER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240501 14:01:08.985970 1519 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240501 14:01:08.986446 1367 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 5 index: 36. Preceding OpId from leader: term: 5 index: 37. (index mismatch)
I20240501 14:01:08.986554 2453 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240501 14:01:08.987092 2453 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240501 14:01:08.987538 2456 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240501 14:01:08.997172 2453 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 5 LEADER]: Committing config change with OpId 5.37: config changed from index 36 to 37, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:08.997715 1367 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:08.999166 1763 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:09.002409 1519 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Committing config change with OpId 5.37: config changed from index 36 to 37, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:09.003194 1261 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 reported cstate change: config changed from index 36 to 37, bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "8d30ece821d04a59962525029d0af320" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:09.009693 1913 consensus_queue.cc:237] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 5.37, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:09.011312 1763 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 5 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20240501 14:01:09.011308 1519 raft_consensus.cc:1261] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Refusing update from remote peer 8d30ece821d04a59962525029d0af320: Log matching property violated. Preceding OpId in replica: term: 5 index: 37. Preceding OpId from leader: term: 5 index: 38. (index mismatch)
I20240501 14:01:09.012027 2435 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240501 14:01:09.012579 2454 consensus_queue.cc:1035] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240501 14:01:09.014341 2454 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 5 LEADER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:09.016950 1310 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 916a3f31691044f783d655695bf0a369 with cas_config_opid_index 37: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.017181 1763 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 FOLLOWER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:09.018532 1267 catalog_manager.cc:5484] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 reported cstate change: config changed from index 37 to 38, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New cstate: current_term: 5 leader_uuid: "8d30ece821d04a59962525029d0af320" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:09.021037 1519 raft_consensus.cc:2932] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Committing config change with OpId 5.38: config changed from index 37 to 38, VOTER c07691f747984bc7a2df956f4f212d8e (127.29.194.193) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } } }
I20240501 14:01:09.025521 1360 tablet_service.cc:1514] Processing DeleteTablet for tablet 916a3f31691044f783d655695bf0a369 with delete_type TABLET_DATA_TOMBSTONED (TS c07691f747984bc7a2df956f4f212d8e not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:58866
I20240501 14:01:09.026890 2463 tablet_replica.cc:330] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:09.027182 2463 raft_consensus.cc:2229] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.027390 2463 raft_consensus.cc:2258] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.028149 2463 ts_tablet_manager.cc:1875] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.033995 2463 ts_tablet_manager.cc:1888] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.37
I20240501 14:01:09.034127 2463 log.cc:1196] T 916a3f31691044f783d655695bf0a369 P c07691f747984bc7a2df956f4f212d8e: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-0-root/wals/916a3f31691044f783d655695bf0a369
I20240501 14:01:09.034837 1252 catalog_manager.cc:4830] TS c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): tablet 916a3f31691044f783d655695bf0a369 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:09.254801 2345 raft_consensus.cc:1050] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: attempting to promote NON_VOTER b1330e34c41745c1bc55375408448018 to VOTER
I20240501 14:01:09.255478 2345 consensus_queue.cc:237] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:09.258016 1839 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 LEARNER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:01:09.258055 1912 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:01:09.258533 1519 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 14:01:09.258648 2465 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:09.259092 2441 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:09.259339 2345 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:09.272814 2441 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:09.273449 1839 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:09.278098 1912 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:09.278821 1259 catalog_manager.cc:5484] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 reported cstate change: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:09.280773 1519 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, b1330e34c41745c1bc55375408448018 (127.29.194.197) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } attrs { replace: true } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:09.284669 1370 consensus_queue.cc:237] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } }
I20240501 14:01:09.286979 1839 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 14:01:09.287477 1912 raft_consensus.cc:1261] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20240501 14:01:09.287513 2450 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:09.288301 2450 consensus_queue.cc:1035] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240501 14:01:09.289851 2441 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:09.290462 1839 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:09.291193 1912 raft_consensus.cc:2932] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } }
I20240501 14:01:09.292996 1252 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:01:09.295276 1259 catalog_manager.cc:5484] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e reported cstate change: config changed from index 33 to 34, VOTER 1e896a0495ca458abad35d883aef4aef (127.29.194.195) evicted. New cstate: current_term: 2 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:09.298167 1508 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c36a1a8c09d415d8c1d1860c0d8b622 with delete_type TABLET_DATA_TOMBSTONED (TS 1e896a0495ca458abad35d883aef4aef not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:56882
I20240501 14:01:09.299026 2469 tablet_replica.cc:330] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:09.299225 2469 raft_consensus.cc:2229] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:09.299479 2469 raft_consensus.cc:2258] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:09.300228 2469 ts_tablet_manager.cc:1875] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:01:09.304068 2469 ts_tablet_manager.cc:1888] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20240501 14:01:09.304150 2469 log.cc:1196] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 1e896a0495ca458abad35d883aef4aef: Deleting WAL directory at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0/minicluster-data/ts-2-root/wals/8c36a1a8c09d415d8c1d1860c0d8b622
I20240501 14:01:09.304754 1253 catalog_manager.cc:4830] TS 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): tablet 8c36a1a8c09d415d8c1d1860c0d8b622 (table test-workload [id=abdbc8974103469498ec20bb03452b7f]) successfully deleted
I20240501 14:01:10.040138 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:10.042642 1370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:01:10.042874 1370 raft_consensus.cc:595] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:01:10.043347 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:10.043570 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:10.315243 2450 raft_consensus.cc:979] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:01:10.315398 2450 raft_consensus.cc:1067] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:01:10.315908 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55950
I20240501 14:01:10.316078 1444 raft_consensus.cc:482] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:10.316164 1444 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:10.317978 1444 raft_consensus.cc:504] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:10.318475 1444 leader_election.cc:288] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 5 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:01:10.318833 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:01:10.318910 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:01:10.319067 1370 raft_consensus.cc:3032] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:10.319118 1519 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:10.319173 1370 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:10.319447 1370 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:10.319737 1370 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:10.321215 1519 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 5.
I20240501 14:01:10.321606 1409 leader_election.cc:302] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:01:10.321660 1370 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 5.
I20240501 14:01:10.321978 2355 raft_consensus.cc:2781] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:10.324955 2355 raft_consensus.cc:686] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:10.325249 2355 consensus_queue.cc:237] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:10.327404 1268 catalog_manager.cc:5484] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 4 to 5, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 5 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:10.722674 1519 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240501 14:01:10.723166 2355 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:10.727479 1370 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240501 14:01:10.728022 2355 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 14:01:12.043998 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:12.046464 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:01:12.046674 1444 raft_consensus.cc:595] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:01:12.047130 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:12.047344 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:12.106659 2355 raft_consensus.cc:979] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:12.106812 2355 raft_consensus.cc:1067] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:12.107388 1370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:58870
I20240501 14:01:12.107556 1370 raft_consensus.cc:482] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:12.107671 1370 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:12.109470 1370 raft_consensus.cc:504] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:12.110008 1370 leader_election.cc:288] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 6 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:01:12.110417 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 31 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:01:12.110437 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 6 candidate_status { last_received { term: 5 index: 31 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:01:12.110656 1519 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:12.110657 1444 raft_consensus.cc:3032] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:01:12.110800 1444 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:12.111068 1444 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:12.111431 1444 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:12.113320 1519 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 6.
I20240501 14:01:12.113725 1334 leader_election.cc:302] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:01:12.113864 1444 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 6.
I20240501 14:01:12.114034 2485 raft_consensus.cc:2781] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:01:12.114337 2485 raft_consensus.cc:686] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 6 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:12.114589 2485 consensus_queue.cc:237] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:12.116657 1259 catalog_manager.cc:5484] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 5 to 6, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 6 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:12.574398 1519 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 6 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240501 14:01:12.574894 2485 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:12.578449 1444 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 6 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 6 index: 32. (index mismatch)
I20240501 14:01:12.578931 2485 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 14:01:14.047758 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:14.050622 1370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:01:14.050840 1370 raft_consensus.cc:595] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 6 LEADER]: Received request to transfer leadership to TS 31d5b9d4a2d2406a972f48086b62ceed
I20240501 14:01:14.051260 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:14.051493 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:14.107555 2497 raft_consensus.cc:979] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:01:14.107740 2497 raft_consensus.cc:1067] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 6 LEADER]: Signalling peer 31d5b9d4a2d2406a972f48086b62ceed to start an election
I20240501 14:01:14.108256 1444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
from {username='slave'} at 127.0.0.1:55950
I20240501 14:01:14.108449 1444 raft_consensus.cc:482] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:14.108554 1444 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:01:14.110432 1444 raft_consensus.cc:504] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:14.110970 1444 leader_election.cc:288] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 7 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049)
I20240501 14:01:14.111377 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 7 candidate_status { last_received { term: 6 index: 32 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:01:14.111377 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3b31989cdae4e9ab72ad37c9e61f903" candidate_uuid: "31d5b9d4a2d2406a972f48086b62ceed" candidate_term: 7 candidate_status { last_received { term: 6 index: 32 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:01:14.111616 1519 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:01:14.111629 1370 raft_consensus.cc:3032] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 6 LEADER]: Stepping down as leader of term 6
I20240501 14:01:14.111748 1370 raft_consensus.cc:729] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:14.112012 1370 consensus_queue.cc:260] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 6.32, Last appended by leader: 32, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:14.112365 1370 raft_consensus.cc:3037] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 6 FOLLOWER]: Advancing to term 7
I20240501 14:01:14.114449 1519 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 7.
I20240501 14:01:14.114921 1370 raft_consensus.cc:2454] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31d5b9d4a2d2406a972f48086b62ceed in term 7.
I20240501 14:01:14.114886 1409 leader_election.cc:302] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, 31d5b9d4a2d2406a972f48086b62ceed; no voters:
I20240501 14:01:14.115214 2355 raft_consensus.cc:2781] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 7 FOLLOWER]: Leader election won for term 7
I20240501 14:01:14.115648 2355 raft_consensus.cc:686] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 7 LEADER]: Becoming Leader. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:14.115952 2355 consensus_queue.cc:237] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 6.32, Last appended by leader: 32, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } }
I20240501 14:01:14.118269 1268 catalog_manager.cc:5484] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed reported cstate change: term changed from 6 to 7, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194). New cstate: current_term: 7 leader_uuid: "31d5b9d4a2d2406a972f48086b62ceed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:14.527382 1519 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 7 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 6 index: 32. Preceding OpId from leader: term: 7 index: 33. (index mismatch)
I20240501 14:01:14.527961 2355 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:14.531710 1370 raft_consensus.cc:1261] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 7 FOLLOWER]: Refusing update from remote peer 31d5b9d4a2d2406a972f48086b62ceed: Log matching property violated. Preceding OpId in replica: term: 6 index: 32. Preceding OpId from leader: term: 7 index: 33. (index mismatch)
I20240501 14:01:14.532250 2355 consensus_queue.cc:1035] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 14:01:16.051831 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:16.054458 1444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
mode: GRACEFUL
new_leader_uuid: "b1330e34c41745c1bc55375408448018"
from {username='slave'} at 127.0.0.1:45496
I20240501 14:01:16.054704 1444 raft_consensus.cc:595] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Received request to transfer leadership to TS b1330e34c41745c1bc55375408448018
I20240501 14:01:16.055105 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:16.055298 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:16.286803 2512 raft_consensus.cc:979] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: : Instructing follower b1330e34c41745c1bc55375408448018 to start an election
I20240501 14:01:16.287035 2511 raft_consensus.cc:1067] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Signalling peer b1330e34c41745c1bc55375408448018 to start an election
I20240501 14:01:16.287789 1839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08"
dest_uuid: "b1330e34c41745c1bc55375408448018"
from {username='slave'} at 127.0.0.1:44040
I20240501 14:01:16.287983 1839 raft_consensus.cc:482] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:16.288092 1839 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:16.290856 1839 raft_consensus.cc:504] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:16.291615 1839 leader_election.cc:288] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [CANDIDATE]: Term 6 election: Requested vote from peers 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379), 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783)
I20240501 14:01:16.292011 1444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "b1330e34c41745c1bc55375408448018" candidate_term: 6 candidate_status { last_received { term: 5 index: 38 } } ignore_live_leader: true dest_uuid: "31d5b9d4a2d2406a972f48086b62ceed"
I20240501 14:01:16.292138 1912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e839ad73564640b24fe9efc9f41f08" candidate_uuid: "b1330e34c41745c1bc55375408448018" candidate_term: 6 candidate_status { last_received { term: 5 index: 38 } } ignore_live_leader: true dest_uuid: "8d30ece821d04a59962525029d0af320"
I20240501 14:01:16.292227 1444 raft_consensus.cc:3032] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:01:16.292356 1444 raft_consensus.cc:729] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 31d5b9d4a2d2406a972f48086b62ceed, State: Running, Role: LEADER
I20240501 14:01:16.292675 1444 consensus_queue.cc:260] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 5.38, Last appended by leader: 38, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:16.293098 1444 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:16.295181 1444 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1330e34c41745c1bc55375408448018 in term 6.
I20240501 14:01:16.295481 1912 raft_consensus.cc:3037] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:16.295608 1805 leader_election.cc:302] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31d5b9d4a2d2406a972f48086b62ceed, b1330e34c41745c1bc55375408448018; no voters:
I20240501 14:01:16.295886 2363 raft_consensus.cc:2781] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:01:16.296255 2363 raft_consensus.cc:686] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 6 LEADER]: Becoming Leader. State: Replica: b1330e34c41745c1bc55375408448018, State: Running, Role: LEADER
I20240501 14:01:16.296527 2363 consensus_queue.cc:237] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 5.38, Last appended by leader: 38, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:16.297801 1912 raft_consensus.cc:2454] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1330e34c41745c1bc55375408448018 in term 6.
I20240501 14:01:16.298982 1259 catalog_manager.cc:5484] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 reported cstate change: term changed from 5 to 6, leader changed from 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194) to b1330e34c41745c1bc55375408448018 (127.29.194.197). New cstate: current_term: 6 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:16.737406 1912 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 6 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 5 index: 38. Preceding OpId from leader: term: 6 index: 39. (index mismatch)
I20240501 14:01:16.738090 2363 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240501 14:01:16.742719 1444 raft_consensus.cc:1261] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 6 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 5 index: 38. Preceding OpId from leader: term: 6 index: 39. (index mismatch)
I20240501 14:01:16.743237 2513 consensus_queue.cc:1035] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31d5b9d4a2d2406a972f48086b62ceed" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 38379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20240501 14:01:18.055680 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:18.069276 1839 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "b1330e34c41745c1bc55375408448018"
mode: GRACEFUL
new_leader_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:38180
I20240501 14:01:18.069527 1839 raft_consensus.cc:595] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 LEADER]: Received request to transfer leadership to TS c07691f747984bc7a2df956f4f212d8e
I20240501 14:01:18.070030 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:18.070256 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:18.474754 2513 raft_consensus.cc:979] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: : Instructing follower c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:18.474907 2513 raft_consensus.cc:1067] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 LEADER]: Signalling peer c07691f747984bc7a2df956f4f212d8e to start an election
I20240501 14:01:18.475526 1370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
from {username='slave'} at 127.0.0.1:45572
I20240501 14:01:18.475725 1370 raft_consensus.cc:482] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:18.475836 1370 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:18.477789 1370 raft_consensus.cc:504] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:18.478447 1370 leader_election.cc:288] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 5 election: Requested vote from peers b1330e34c41745c1bc55375408448018 (127.29.194.197:39411), 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783)
I20240501 14:01:18.478865 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "b1330e34c41745c1bc55375408448018"
I20240501 14:01:18.478866 1913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "c07691f747984bc7a2df956f4f212d8e" candidate_term: 5 candidate_status { last_received { term: 4 index: 34 } } ignore_live_leader: true dest_uuid: "8d30ece821d04a59962525029d0af320"
I20240501 14:01:18.479105 1839 raft_consensus.cc:3032] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 LEADER]: Stepping down as leader of term 4
I20240501 14:01:18.479111 1913 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:18.479228 1839 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b1330e34c41745c1bc55375408448018, State: Running, Role: LEADER
I20240501 14:01:18.479503 1839 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:18.479892 1839 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Advancing to term 5
I20240501 14:01:18.481992 1913 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 5.
I20240501 14:01:18.482411 1839 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c07691f747984bc7a2df956f4f212d8e in term 5.
I20240501 14:01:18.482424 1333 leader_election.cc:302] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d30ece821d04a59962525029d0af320, c07691f747984bc7a2df956f4f212d8e; no voters:
I20240501 14:01:18.482754 2497 raft_consensus.cc:2781] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Leader election won for term 5
I20240501 14:01:18.483125 2497 raft_consensus.cc:686] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 5 LEADER]: Becoming Leader. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:18.483381 2497 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 34, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:18.485792 1268 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e reported cstate change: term changed from 4 to 5, leader changed from b1330e34c41745c1bc55375408448018 (127.29.194.197) to c07691f747984bc7a2df956f4f212d8e (127.29.194.193). New cstate: current_term: 5 leader_uuid: "c07691f747984bc7a2df956f4f212d8e" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:18.971021 1913 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240501 14:01:18.971616 2497 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:18.975245 1839 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Refusing update from remote peer c07691f747984bc7a2df956f4f212d8e: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 5 index: 35. (index mismatch)
I20240501 14:01:18.975800 2497 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240501 14:01:20.070582 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:20.073297 1370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
mode: GRACEFUL
new_leader_uuid: "b1330e34c41745c1bc55375408448018"
from {username='slave'} at 127.0.0.1:45586
I20240501 14:01:20.073555 1370 raft_consensus.cc:595] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 5 LEADER]: Received request to transfer leadership to TS b1330e34c41745c1bc55375408448018
I20240501 14:01:20.074018 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:20.074194 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:20.453518 2544 raft_consensus.cc:979] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: : Instructing follower b1330e34c41745c1bc55375408448018 to start an election
I20240501 14:01:20.453701 2544 raft_consensus.cc:1067] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 5 LEADER]: Signalling peer b1330e34c41745c1bc55375408448018 to start an election
I20240501 14:01:20.454346 1839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69"
dest_uuid: "b1330e34c41745c1bc55375408448018"
from {username='slave'} at 127.0.0.1:44020
I20240501 14:01:20.454551 1839 raft_consensus.cc:482] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:20.454656 1839 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:20.456601 1839 raft_consensus.cc:504] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:20.457345 1839 leader_election.cc:288] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [CANDIDATE]: Term 6 election: Requested vote from peers c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821), 8d30ece821d04a59962525029d0af320 (127.29.194.198:35783)
I20240501 14:01:20.457823 1370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "b1330e34c41745c1bc55375408448018" candidate_term: 6 candidate_status { last_received { term: 5 index: 35 } } ignore_live_leader: true dest_uuid: "c07691f747984bc7a2df956f4f212d8e"
I20240501 14:01:20.457927 1913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ee8e1d3ee3b429ebd15a32afe393e69" candidate_uuid: "b1330e34c41745c1bc55375408448018" candidate_term: 6 candidate_status { last_received { term: 5 index: 35 } } ignore_live_leader: true dest_uuid: "8d30ece821d04a59962525029d0af320"
I20240501 14:01:20.458019 1370 raft_consensus.cc:3032] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 5 LEADER]: Stepping down as leader of term 5
I20240501 14:01:20.458108 1370 raft_consensus.cc:729] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c07691f747984bc7a2df956f4f212d8e, State: Running, Role: LEADER
I20240501 14:01:20.458140 1913 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:20.458396 1370 consensus_queue.cc:260] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:20.458736 1370 raft_consensus.cc:3037] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Advancing to term 6
I20240501 14:01:20.460333 1913 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1330e34c41745c1bc55375408448018 in term 6.
I20240501 14:01:20.460803 1802 leader_election.cc:302] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d30ece821d04a59962525029d0af320, b1330e34c41745c1bc55375408448018; no voters:
I20240501 14:01:20.460909 1370 raft_consensus.cc:2454] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1330e34c41745c1bc55375408448018 in term 6.
I20240501 14:01:20.461268 2513 raft_consensus.cc:2781] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 6 FOLLOWER]: Leader election won for term 6
I20240501 14:01:20.461683 2513 raft_consensus.cc:686] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 6 LEADER]: Becoming Leader. State: Replica: b1330e34c41745c1bc55375408448018, State: Running, Role: LEADER
I20240501 14:01:20.461944 2513 consensus_queue.cc:237] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 5.35, Last appended by leader: 35, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } }
I20240501 14:01:20.464501 1259 catalog_manager.cc:5484] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 reported cstate change: term changed from 5 to 6, leader changed from c07691f747984bc7a2df956f4f212d8e (127.29.194.193) to b1330e34c41745c1bc55375408448018 (127.29.194.197). New cstate: current_term: 6 leader_uuid: "b1330e34c41745c1bc55375408448018" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:21.007639 1913 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 6 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 6 index: 36. (index mismatch)
I20240501 14:01:21.008176 2513 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d30ece821d04a59962525029d0af320" member_type: VOTER last_known_addr { host: "127.29.194.198" port: 35783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 14:01:21.012069 1370 raft_consensus.cc:1261] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 6 FOLLOWER]: Refusing update from remote peer b1330e34c41745c1bc55375408448018: Log matching property violated. Preceding OpId in replica: term: 5 index: 35. Preceding OpId from leader: term: 6 index: 36. (index mismatch)
I20240501 14:01:21.012856 2552 consensus_queue.cc:1035] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c07691f747984bc7a2df956f4f212d8e" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 46821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240501 14:01:22.074625 1321 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 14:01:22.077219 1839 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41"
dest_uuid: "b1330e34c41745c1bc55375408448018"
mode: GRACEFUL
new_leader_uuid: "bf66d98ce8a34944be110a6fa9c75d09"
from {username='slave'} at 127.0.0.1:38180
I20240501 14:01:22.077426 1839 raft_consensus.cc:595] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 LEADER]: Received request to transfer leadership to TS bf66d98ce8a34944be110a6fa9c75d09
I20240501 14:01:22.077906 1321 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:22.078150 1321 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:22.186619 2513 raft_consensus.cc:979] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: : Instructing follower bf66d98ce8a34944be110a6fa9c75d09 to start an election
I20240501 14:01:22.186775 2513 raft_consensus.cc:1067] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 LEADER]: Signalling peer bf66d98ce8a34944be110a6fa9c75d09 to start an election
I20240501 14:01:22.187350 1762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41"
dest_uuid: "bf66d98ce8a34944be110a6fa9c75d09"
from {username='slave'} at 127.0.0.1:33434
I20240501 14:01:22.187527 1762 raft_consensus.cc:482] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:22.187630 1762 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:22.189575 1762 raft_consensus.cc:504] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:22.190425 1762 leader_election.cc:288] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [CANDIDATE]: Term 4 election: Requested vote from peers 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049), b1330e34c41745c1bc55375408448018 (127.29.194.197:39411)
I20240501 14:01:22.190814 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "bf66d98ce8a34944be110a6fa9c75d09" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: true dest_uuid: "b1330e34c41745c1bc55375408448018"
I20240501 14:01:22.190785 1519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2d33db85414876998a3ed550e44e41" candidate_uuid: "bf66d98ce8a34944be110a6fa9c75d09" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: true dest_uuid: "1e896a0495ca458abad35d883aef4aef"
I20240501 14:01:22.191032 1839 raft_consensus.cc:3032] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 LEADER]: Stepping down as leader of term 3
I20240501 14:01:22.191042 1519 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:22.191156 1839 raft_consensus.cc:729] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b1330e34c41745c1bc55375408448018, State: Running, Role: LEADER
I20240501 14:01:22.191432 1839 consensus_queue.cc:260] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 3.36, Last appended by leader: 36, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:22.191797 1839 raft_consensus.cc:3037] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 3 FOLLOWER]: Advancing to term 4
I20240501 14:01:22.194021 1519 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf66d98ce8a34944be110a6fa9c75d09 in term 4.
I20240501 14:01:22.194422 1839 raft_consensus.cc:2454] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf66d98ce8a34944be110a6fa9c75d09 in term 4.
I20240501 14:01:22.194473 1728 leader_election.cc:302] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e896a0495ca458abad35d883aef4aef, bf66d98ce8a34944be110a6fa9c75d09; no voters:
I20240501 14:01:22.194798 2543 raft_consensus.cc:2781] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 4 FOLLOWER]: Leader election won for term 4
I20240501 14:01:22.195197 2543 raft_consensus.cc:686] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 4 LEADER]: Becoming Leader. State: Replica: bf66d98ce8a34944be110a6fa9c75d09, State: Running, Role: LEADER
I20240501 14:01:22.195478 2543 consensus_queue.cc:237] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 3.36, Last appended by leader: 36, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } }
I20240501 14:01:22.197897 1268 catalog_manager.cc:5484] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 reported cstate change: term changed from 3 to 4, leader changed from b1330e34c41745c1bc55375408448018 (127.29.194.197) to bf66d98ce8a34944be110a6fa9c75d09 (127.29.194.196). New cstate: current_term: 4 leader_uuid: "bf66d98ce8a34944be110a6fa9c75d09" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf66d98ce8a34944be110a6fa9c75d09" member_type: VOTER last_known_addr { host: "127.29.194.196" port: 41675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:01:22.591917 1519 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 3 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240501 14:01:22.592411 2543 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e896a0495ca458abad35d883aef4aef" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240501 14:01:22.596012 1839 raft_consensus.cc:1261] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Refusing update from remote peer bf66d98ce8a34944be110a6fa9c75d09: Log matching property violated. Preceding OpId in replica: term: 3 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240501 14:01:22.596532 2543 consensus_queue.cc:1035] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1330e34c41745c1bc55375408448018" member_type: VOTER last_known_addr { host: "127.29.194.197" port: 39411 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:195: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:395: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240501 14:01:22.634608 30475 tablet_server.cc:178] TabletServer@127.29.194.193:0 shutting down...
W20240501 14:01:22.653512 1802 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20240501 14:01:22.654937 1802 consensus_peers.cc:485] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 -> Peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): Couldn't send request to peer c07691f747984bc7a2df956f4f212d8e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.658131 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:01:22.658453 30475 tablet_replica.cc:330] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:22.658653 30475 raft_consensus.cc:2229] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.658934 30475 raft_consensus.cc:2258] T 8c36a1a8c09d415d8c1d1860c0d8b622 P c07691f747984bc7a2df956f4f212d8e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.659518 30475 tablet_replica.cc:330] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:22.659653 30475 raft_consensus.cc:2229] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 3 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.659909 30475 raft_consensus.cc:2258] T 348221e7dbd04efcba6185ba29a33a60 P c07691f747984bc7a2df956f4f212d8e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.660378 30475 tablet_replica.cc:330] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:22.660478 30475 raft_consensus.cc:2229] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.660585 30475 raft_consensus.cc:2258] T 407fed285f274f3c95342a21421b2c66 P c07691f747984bc7a2df956f4f212d8e [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.661002 30475 tablet_replica.cc:330] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:22.661095 30475 raft_consensus.cc:2229] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.661262 30475 raft_consensus.cc:2258] T 2bdda4e923dc4caaba534f1e8b5693e2 P c07691f747984bc7a2df956f4f212d8e [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.661633 30475 tablet_replica.cc:330] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:22.661733 30475 raft_consensus.cc:2229] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 7 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.661841 30475 raft_consensus.cc:2258] T d3b31989cdae4e9ab72ad37c9e61f903 P c07691f747984bc7a2df956f4f212d8e [term 7 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.662235 30475 tablet_replica.cc:330] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e: stopping tablet replica
I20240501 14:01:22.662345 30475 raft_consensus.cc:2229] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 6 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.662451 30475 raft_consensus.cc:2258] T 6ee8e1d3ee3b429ebd15a32afe393e69 P c07691f747984bc7a2df956f4f212d8e [term 6 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.672183 30475 tablet_server.cc:194] TabletServer@127.29.194.193:0 shutdown complete.
I20240501 14:01:22.677799 30475 tablet_server.cc:178] TabletServer@127.29.194.194:0 shutting down...
W20240501 14:01:22.679234 1408 consensus_peers.cc:485] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed -> Peer c07691f747984bc7a2df956f4f212d8e (127.29.194.193:46821): Couldn't send request to peer c07691f747984bc7a2df956f4f212d8e. Status: Network error: Client connection negotiation failed: client connection to 127.29.194.193:46821: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.687887 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:01:22.688180 30475 tablet_replica.cc:330] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:22.688352 30475 raft_consensus.cc:2229] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.689067 30475 raft_consensus.cc:2258] T 2bdda4e923dc4caaba534f1e8b5693e2 P 31d5b9d4a2d2406a972f48086b62ceed [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.689669 30475 tablet_replica.cc:330] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:22.689813 30475 raft_consensus.cc:2229] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.689941 30475 raft_consensus.cc:2258] T 348221e7dbd04efcba6185ba29a33a60 P 31d5b9d4a2d2406a972f48086b62ceed [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.690385 30475 tablet_replica.cc:330] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:22.690526 30475 raft_consensus.cc:2229] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.690730 30475 raft_consensus.cc:2258] T 407fed285f274f3c95342a21421b2c66 P 31d5b9d4a2d2406a972f48086b62ceed [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.691135 30475 tablet_replica.cc:330] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:22.691226 30475 raft_consensus.cc:2229] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 6 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.691311 30475 raft_consensus.cc:2258] T 78e839ad73564640b24fe9efc9f41f08 P 31d5b9d4a2d2406a972f48086b62ceed [term 6 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.691653 30475 tablet_replica.cc:330] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:22.691738 30475 raft_consensus.cc:2229] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.691915 30475 raft_consensus.cc:2258] T 355ae59edd1c4620904ba23275021582 P 31d5b9d4a2d2406a972f48086b62ceed [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.692277 30475 tablet_replica.cc:330] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed: stopping tablet replica
I20240501 14:01:22.692364 30475 raft_consensus.cc:2229] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 7 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.692565 30475 raft_consensus.cc:2258] T d3b31989cdae4e9ab72ad37c9e61f903 P 31d5b9d4a2d2406a972f48086b62ceed [term 7 FOLLOWER]: Raft consensus is shut down!
W20240501 14:01:22.705804 1730 consensus_peers.cc:485] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 -> Peer 31d5b9d4a2d2406a972f48086b62ceed (127.29.194.194:38379): Couldn't send request to peer 31d5b9d4a2d2406a972f48086b62ceed. Status: Network error: Client connection negotiation failed: client connection to 127.29.194.194:38379: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.723356 30475 tablet_server.cc:194] TabletServer@127.29.194.194:0 shutdown complete.
I20240501 14:01:22.728884 30475 tablet_server.cc:178] TabletServer@127.29.194.195:0 shutting down...
I20240501 14:01:22.737879 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:01:22.738121 30475 tablet_replica.cc:330] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:22.738257 30475 raft_consensus.cc:2229] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 5 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.738533 30475 raft_consensus.cc:2258] T 407fed285f274f3c95342a21421b2c66 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.739104 30475 tablet_replica.cc:330] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:22.739228 30475 raft_consensus.cc:2229] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.739436 30475 raft_consensus.cc:2258] T 7ca80506974245cb887559f7f884fb98 P 1e896a0495ca458abad35d883aef4aef [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.739873 30475 tablet_replica.cc:330] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:22.740025 30475 raft_consensus.cc:2229] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 7 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.740186 30475 raft_consensus.cc:2258] T d3b31989cdae4e9ab72ad37c9e61f903 P 1e896a0495ca458abad35d883aef4aef [term 7 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.740767 30475 tablet_replica.cc:330] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:22.740940 30475 raft_consensus.cc:2229] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.741186 30475 raft_consensus.cc:2258] T 916a3f31691044f783d655695bf0a369 P 1e896a0495ca458abad35d883aef4aef [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.741678 30475 tablet_replica.cc:330] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:22.741813 30475 raft_consensus.cc:2229] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.742163 30475 raft_consensus.cc:2258] T ae2d33db85414876998a3ed550e44e41 P 1e896a0495ca458abad35d883aef4aef [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.742798 30475 tablet_replica.cc:330] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef: stopping tablet replica
I20240501 14:01:22.742918 30475 raft_consensus.cc:2229] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.743117 30475 raft_consensus.cc:2258] T 46ae855dc1a44159b5f3cca0657dc752 P 1e896a0495ca458abad35d883aef4aef [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.749784 30475 tablet_server.cc:194] TabletServer@127.29.194.195:0 shutdown complete.
I20240501 14:01:22.754650 30475 tablet_server.cc:178] TabletServer@127.29.194.196:0 shutting down...
I20240501 14:01:22.780143 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:01:22.780416 30475 tablet_replica.cc:330] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09: stopping tablet replica
I20240501 14:01:22.780620 30475 raft_consensus.cc:2229] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.780916 30475 raft_consensus.cc:2258] T 46ae855dc1a44159b5f3cca0657dc752 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.781558 30475 tablet_replica.cc:330] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09: stopping tablet replica
I20240501 14:01:22.781709 30475 raft_consensus.cc:2229] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 4 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.781991 30475 raft_consensus.cc:2258] T ae2d33db85414876998a3ed550e44e41 P bf66d98ce8a34944be110a6fa9c75d09 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.782655 30475 tablet_replica.cc:330] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09: stopping tablet replica
I20240501 14:01:22.782791 30475 raft_consensus.cc:2229] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.782967 30475 raft_consensus.cc:2258] T 916a3f31691044f783d655695bf0a369 P bf66d98ce8a34944be110a6fa9c75d09 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.783368 30475 tablet_replica.cc:330] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09: stopping tablet replica
I20240501 14:01:22.783492 30475 raft_consensus.cc:2229] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.783747 30475 raft_consensus.cc:2258] T 7ca80506974245cb887559f7f884fb98 P bf66d98ce8a34944be110a6fa9c75d09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.784133 30475 tablet_replica.cc:330] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09: stopping tablet replica
I20240501 14:01:22.784256 30475 raft_consensus.cc:2229] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.784502 30475 raft_consensus.cc:2258] T 355ae59edd1c4620904ba23275021582 P bf66d98ce8a34944be110a6fa9c75d09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.784950 30475 tablet_replica.cc:330] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09: stopping tablet replica
I20240501 14:01:22.785087 30475 raft_consensus.cc:2229] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.785207 30475 raft_consensus.cc:2258] T 348221e7dbd04efcba6185ba29a33a60 P bf66d98ce8a34944be110a6fa9c75d09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.789945 30475 tablet_server.cc:194] TabletServer@127.29.194.196:0 shutdown complete.
I20240501 14:01:22.793702 30475 tablet_server.cc:178] TabletServer@127.29.194.197:0 shutting down...
I20240501 14:01:22.802978 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:01:22.803205 30475 tablet_replica.cc:330] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018: stopping tablet replica
I20240501 14:01:22.803349 30475 raft_consensus.cc:2229] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.803539 30475 raft_consensus.cc:2258] T 355ae59edd1c4620904ba23275021582 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.804015 30475 tablet_replica.cc:330] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018: stopping tablet replica
I20240501 14:01:22.804126 30475 raft_consensus.cc:2229] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.804270 30475 raft_consensus.cc:2258] T 8c36a1a8c09d415d8c1d1860c0d8b622 P b1330e34c41745c1bc55375408448018 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.804646 30475 tablet_replica.cc:330] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018: stopping tablet replica
I20240501 14:01:22.804812 30475 raft_consensus.cc:2229] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.804960 30475 raft_consensus.cc:2258] T 2bdda4e923dc4caaba534f1e8b5693e2 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.805349 30475 tablet_replica.cc:330] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018: stopping tablet replica
I20240501 14:01:22.805445 30475 raft_consensus.cc:2229] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 6 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.805657 30475 raft_consensus.cc:2258] T 78e839ad73564640b24fe9efc9f41f08 P b1330e34c41745c1bc55375408448018 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.806067 30475 tablet_replica.cc:330] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018: stopping tablet replica
I20240501 14:01:22.806162 30475 raft_consensus.cc:2229] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.806255 30475 raft_consensus.cc:2258] T ae2d33db85414876998a3ed550e44e41 P b1330e34c41745c1bc55375408448018 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.806773 30475 tablet_replica.cc:330] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018: stopping tablet replica
I20240501 14:01:22.806864 30475 raft_consensus.cc:2229] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 6 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.807089 30475 raft_consensus.cc:2258] T 6ee8e1d3ee3b429ebd15a32afe393e69 P b1330e34c41745c1bc55375408448018 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.821743 30475 tablet_server.cc:194] TabletServer@127.29.194.197:0 shutdown complete.
I20240501 14:01:22.828635 30475 tablet_server.cc:178] TabletServer@127.29.194.198:0 shutting down...
W20240501 14:01:22.834722 1877 consensus_peers.cc:485] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 -> Peer 1e896a0495ca458abad35d883aef4aef (127.29.194.195:38049): Couldn't send request to peer 1e896a0495ca458abad35d883aef4aef. Status: Network error: Client connection negotiation failed: client connection to 127.29.194.195:38049: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:22.839226 30475 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 14:01:22.839481 30475 tablet_replica.cc:330] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320: stopping tablet replica
I20240501 14:01:22.839673 30475 raft_consensus.cc:2229] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.839874 30475 raft_consensus.cc:2258] T 78e839ad73564640b24fe9efc9f41f08 P 8d30ece821d04a59962525029d0af320 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.840368 30475 tablet_replica.cc:330] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320: stopping tablet replica
I20240501 14:01:22.840482 30475 raft_consensus.cc:2229] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.840638 30475 raft_consensus.cc:2258] T 8c36a1a8c09d415d8c1d1860c0d8b622 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.841082 30475 tablet_replica.cc:330] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320: stopping tablet replica
I20240501 14:01:22.841182 30475 raft_consensus.cc:2229] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 2 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.841388 30475 raft_consensus.cc:2258] T 46ae855dc1a44159b5f3cca0657dc752 P 8d30ece821d04a59962525029d0af320 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.841787 30475 tablet_replica.cc:330] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320: stopping tablet replica
I20240501 14:01:22.841897 30475 raft_consensus.cc:2229] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.842036 30475 raft_consensus.cc:2258] T 7ca80506974245cb887559f7f884fb98 P 8d30ece821d04a59962525029d0af320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.842388 30475 tablet_replica.cc:330] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320: stopping tablet replica
I20240501 14:01:22.842479 30475 raft_consensus.cc:2229] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 6 FOLLOWER]: Raft consensus shutting down.
I20240501 14:01:22.842653 30475 raft_consensus.cc:2258] T 6ee8e1d3ee3b429ebd15a32afe393e69 P 8d30ece821d04a59962525029d0af320 [term 6 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.843008 30475 tablet_replica.cc:330] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320: stopping tablet replica
I20240501 14:01:22.843097 30475 raft_consensus.cc:2229] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 5 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.843286 30475 raft_consensus.cc:2258] T 916a3f31691044f783d655695bf0a369 P 8d30ece821d04a59962525029d0af320 [term 5 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.857663 30475 tablet_server.cc:194] TabletServer@127.29.194.198:0 shutdown complete.
I20240501 14:01:22.861689 30475 master.cc:537] Master@127.29.194.254:37029 shutting down...
I20240501 14:01:22.883097 30475 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 1 LEADER]: Raft consensus shutting down.
I20240501 14:01:22.883344 30475 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:01:22.883447 30475 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1e2f523513904e35ad8e2accc3750429: stopping tablet replica
I20240501 14:01:22.897305 30475 master.cc:552] Master@127.29.194.254:37029 shutdown complete.
I20240501 14:01:22.904865 30475 test_util.cc:175] -----------------------------------------------
I20240501 14:01:22.904927 30475 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714571949891039-30475-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (37012 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20240501 14:01:22.906673 30475 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.194.254:35515
I20240501 14:01:22.907119 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:22.909438 2577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:22.910158 2578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:22.910837 2580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:22.911417 30475 server_base.cc:1034] running on GCE node
I20240501 14:01:22.911641 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:01:22.911695 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:01:22.911731 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572082911727 us; error 0 us; skew 500 ppm
I20240501 14:01:22.911875 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:22.913213 30475 webserver.cc:460] Webserver started at http://127.29.194.254:42675/ using document root <none> and password file <none>
I20240501 14:01:22.913437 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:22.913497 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:22.913640 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:22.914274 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/master-0-root/instance:
uuid: "a038393e0dc54aeeaaa32e0e31f112a3"
format_stamp: "Formatted at 2024-05-01 14:01:22 on dist-test-slave-kwk7"
I20240501 14:01:22.916771 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:01:22.918252 2585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:22.918596 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:01:22.918700 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/master-0-root
uuid: "a038393e0dc54aeeaaa32e0e31f112a3"
format_stamp: "Formatted at 2024-05-01 14:01:22 on dist-test-slave-kwk7"
I20240501 14:01:22.918814 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:22.939415 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:22.939929 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:22.949950 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.254:35515
I20240501 14:01:22.949966 2636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.254:35515 every 8 connection(s)
I20240501 14:01:22.951256 2637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:22.955258 2637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3: Bootstrap starting.
I20240501 14:01:22.956422 2637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:22.958014 2637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3: No bootstrap required, opened a new log
I20240501 14:01:22.958559 2637 raft_consensus.cc:348] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a038393e0dc54aeeaaa32e0e31f112a3" member_type: VOTER }
I20240501 14:01:22.958698 2637 raft_consensus.cc:374] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:22.958748 2637 raft_consensus.cc:729] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a038393e0dc54aeeaaa32e0e31f112a3, State: Initialized, Role: FOLLOWER
I20240501 14:01:22.958945 2637 consensus_queue.cc:260] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a038393e0dc54aeeaaa32e0e31f112a3" member_type: VOTER }
I20240501 14:01:22.959059 2637 raft_consensus.cc:388] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:01:22.959115 2637 raft_consensus.cc:482] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:01:22.959182 2637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:22.960932 2637 raft_consensus.cc:504] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a038393e0dc54aeeaaa32e0e31f112a3" member_type: VOTER }
I20240501 14:01:22.961112 2637 leader_election.cc:302] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a038393e0dc54aeeaaa32e0e31f112a3; no voters:
I20240501 14:01:22.961489 2637 leader_election.cc:288] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:01:22.961584 2640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:22.962062 2640 raft_consensus.cc:686] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [term 1 LEADER]: Becoming Leader. State: Replica: a038393e0dc54aeeaaa32e0e31f112a3, State: Running, Role: LEADER
I20240501 14:01:22.962402 2637 sys_catalog.cc:564] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:01:22.962368 2640 consensus_queue.cc:237] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a038393e0dc54aeeaaa32e0e31f112a3" member_type: VOTER }
I20240501 14:01:22.963423 2641 sys_catalog.cc:455] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a038393e0dc54aeeaaa32e0e31f112a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a038393e0dc54aeeaaa32e0e31f112a3" member_type: VOTER } }
I20240501 14:01:22.963490 2642 sys_catalog.cc:455] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a038393e0dc54aeeaaa32e0e31f112a3. Latest consensus state: current_term: 1 leader_uuid: "a038393e0dc54aeeaaa32e0e31f112a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a038393e0dc54aeeaaa32e0e31f112a3" member_type: VOTER } }
I20240501 14:01:22.963721 2641 sys_catalog.cc:458] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [sys.catalog]: This master's current role is: LEADER
I20240501 14:01:22.963758 2642 sys_catalog.cc:458] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3 [sys.catalog]: This master's current role is: LEADER
I20240501 14:01:22.964824 2646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:01:22.966009 2646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:01:22.967640 30475 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 14:01:22.968858 2646 catalog_manager.cc:1347] Generated new cluster ID: e709425bb4dc4f89be329913fe4903a0
I20240501 14:01:22.968935 2646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:01:22.980624 2646 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:01:22.981475 2646 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:01:22.991232 2646 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a038393e0dc54aeeaaa32e0e31f112a3: Generated new TSK 0
I20240501 14:01:22.991498 2646 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:01:23.000229 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:23.003125 2658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:23.003932 2659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:23.004592 2661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:23.005244 30475 server_base.cc:1034] running on GCE node
I20240501 14:01:23.005450 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:01:23.005497 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:01:23.005529 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572083005526 us; error 0 us; skew 500 ppm
I20240501 14:01:23.005631 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:23.006858 30475 webserver.cc:460] Webserver started at http://127.29.194.193:34337/ using document root <none> and password file <none>
I20240501 14:01:23.007073 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:23.007130 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:23.007225 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:23.007819 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-0-root/instance:
uuid: "4a078c6056e74c4594b6fab2049c1b70"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-kwk7"
I20240501 14:01:23.010332 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:23.011826 2666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.012163 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:01:23.012259 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-0-root
uuid: "4a078c6056e74c4594b6fab2049c1b70"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-kwk7"
I20240501 14:01:23.012367 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:23.030738 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:23.031275 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:23.031939 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:23.032876 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:23.032936 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.033010 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:23.033048 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.045722 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.193:39059
I20240501 14:01:23.045734 2728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.193:39059 every 8 connection(s)
I20240501 14:01:23.047400 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:23.050388 2733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:23.051110 2734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:23.051801 2736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:23.052033 30475 server_base.cc:1034] running on GCE node
I20240501 14:01:23.052256 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:01:23.052306 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:01:23.052348 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572083052344 us; error 0 us; skew 500 ppm
I20240501 14:01:23.052508 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:23.054196 2729 heartbeater.cc:347] Connected to a master server at 127.29.194.254:35515
I20240501 14:01:23.054231 30475 webserver.cc:460] Webserver started at http://127.29.194.194:40815/ using document root <none> and password file <none>
I20240501 14:01:23.054343 2729 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:23.054464 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:23.054543 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:23.054669 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:23.054759 2729 heartbeater.cc:511] Master 127.29.194.254:35515 requested a full tablet report, sending...
I20240501 14:01:23.055359 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-1-root/instance:
uuid: "cc5ec466352142ce9f1b281c5e2784c3"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-kwk7"
I20240501 14:01:23.055611 2602 ts_manager.cc:194] Registered new tserver with Master: 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193:39059)
I20240501 14:01:23.056629 2602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54072
I20240501 14:01:23.057984 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:01:23.059458 2741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.059774 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:01:23.059856 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-1-root
uuid: "cc5ec466352142ce9f1b281c5e2784c3"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-kwk7"
I20240501 14:01:23.059958 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:23.095062 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:23.095607 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:23.096165 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:23.096987 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:23.097054 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.097111 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:23.097155 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.110669 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.194:41969
I20240501 14:01:23.110702 2803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.194:41969 every 8 connection(s)
I20240501 14:01:23.112146 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:23.115242 2807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:23.116209 2808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:23.116269 30475 server_base.cc:1034] running on GCE node
W20240501 14:01:23.117022 2810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:23.117316 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:01:23.117385 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:01:23.117416 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572083117412 us; error 0 us; skew 500 ppm
I20240501 14:01:23.117555 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:23.118813 2804 heartbeater.cc:347] Connected to a master server at 127.29.194.254:35515
I20240501 14:01:23.118896 30475 webserver.cc:460] Webserver started at http://127.29.194.195:35457/ using document root <none> and password file <none>
I20240501 14:01:23.118970 2804 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:23.119117 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:23.119190 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:23.119308 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:23.119378 2804 heartbeater.cc:511] Master 127.29.194.254:35515 requested a full tablet report, sending...
I20240501 14:01:23.120003 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-2-root/instance:
uuid: "047859656efe4f979dbeb8e5bec3ad32"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-kwk7"
I20240501 14:01:23.120251 2602 ts_manager.cc:194] Registered new tserver with Master: cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969)
I20240501 14:01:23.121137 2602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54076
I20240501 14:01:23.122593 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 14:01:23.124049 2815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.124347 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20240501 14:01:23.124436 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-2-root
uuid: "047859656efe4f979dbeb8e5bec3ad32"
format_stamp: "Formatted at 2024-05-01 14:01:23 on dist-test-slave-kwk7"
I20240501 14:01:23.124547 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:23.138532 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:23.138994 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:23.139523 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:23.140306 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:23.140358 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.140424 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:23.140457 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:23.154502 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.195:39181
I20240501 14:01:23.154532 2877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.195:39181 every 8 connection(s)
I20240501 14:01:23.160909 2878 heartbeater.cc:347] Connected to a master server at 127.29.194.254:35515
I20240501 14:01:23.161037 2878 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:23.161366 2878 heartbeater.cc:511] Master 127.29.194.254:35515 requested a full tablet report, sending...
I20240501 14:01:23.162202 2602 ts_manager.cc:194] Registered new tserver with Master: 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
I20240501 14:01:23.162288 30475 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006903892s
I20240501 14:01:23.163283 2602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54080
I20240501 14:01:24.057881 2729 heartbeater.cc:503] Master 127.29.194.254:35515 was elected leader, sending a full tablet report...
I20240501 14:01:24.122375 2804 heartbeater.cc:503] Master 127.29.194.254:35515 was elected leader, sending a full tablet report...
I20240501 14:01:24.164562 2878 heartbeater.cc:503] Master 127.29.194.254:35515 was elected leader, sending a full tablet report...
I20240501 14:01:24.189868 30475 test_util.cc:268] Using random seed: -335209820
I20240501 14:01:24.202535 2602 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54082:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 14:01:24.203222 2602 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 14:01:24.237277 2769 tablet_service.cc:1467] Processing CreateTablet for tablet a6ffa91a5dd84bc8a7b09bd9a612595a (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 14:01:24.237284 2768 tablet_service.cc:1467] Processing CreateTablet for tablet baa36559e072427bafab53108b0d4dd2 (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 14:01:24.237284 2767 tablet_service.cc:1467] Processing CreateTablet for tablet 302ae30a496f4ae79492fa6388e916ad (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 14:01:24.237762 2694 tablet_service.cc:1467] Processing CreateTablet for tablet a6ffa91a5dd84bc8a7b09bd9a612595a (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 14:01:24.238076 2769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ffa91a5dd84bc8a7b09bd9a612595a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.238015 2843 tablet_service.cc:1467] Processing CreateTablet for tablet a6ffa91a5dd84bc8a7b09bd9a612595a (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 14:01:24.238385 2843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ffa91a5dd84bc8a7b09bd9a612595a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.238685 2692 tablet_service.cc:1467] Processing CreateTablet for tablet 302ae30a496f4ae79492fa6388e916ad (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 14:01:24.239138 2767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302ae30a496f4ae79492fa6388e916ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.240927 2768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa36559e072427bafab53108b0d4dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.238394 2693 tablet_service.cc:1467] Processing CreateTablet for tablet baa36559e072427bafab53108b0d4dd2 (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 14:01:24.238111 2694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ffa91a5dd84bc8a7b09bd9a612595a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.245992 2692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302ae30a496f4ae79492fa6388e916ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.248955 2693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa36559e072427bafab53108b0d4dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.253223 2900 tablet_bootstrap.cc:492] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70: Bootstrap starting.
I20240501 14:01:24.253751 2899 tablet_bootstrap.cc:492] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3: Bootstrap starting.
I20240501 14:01:24.255134 2900 tablet_bootstrap.cc:654] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.255163 2899 tablet_bootstrap.cc:654] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.256319 2842 tablet_service.cc:1467] Processing CreateTablet for tablet baa36559e072427bafab53108b0d4dd2 (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 14:01:24.256333 2841 tablet_service.cc:1467] Processing CreateTablet for tablet 302ae30a496f4ae79492fa6388e916ad (DEFAULT_TABLE table=test-workload [id=3d7ded2460d84eafa21318b33ece48a1]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 14:01:24.256868 2842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa36559e072427bafab53108b0d4dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.257086 2841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302ae30a496f4ae79492fa6388e916ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:24.259891 2898 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32: Bootstrap starting.
I20240501 14:01:24.261627 2898 tablet_bootstrap.cc:654] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.263254 2900 tablet_bootstrap.cc:492] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70: No bootstrap required, opened a new log
I20240501 14:01:24.263429 2900 ts_tablet_manager.cc:1367] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.010s
I20240501 14:01:24.264145 2900 raft_consensus.cc:348] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.264446 2900 raft_consensus.cc:374] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.264555 2900 raft_consensus.cc:729] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a078c6056e74c4594b6fab2049c1b70, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.264997 2900 consensus_queue.cc:260] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.268014 2900 ts_tablet_manager.cc:1398] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:01:24.268712 2900 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70: Bootstrap starting.
I20240501 14:01:24.271708 2900 tablet_bootstrap.cc:654] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.274188 2898 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32: No bootstrap required, opened a new log
I20240501 14:01:24.274188 2900 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70: No bootstrap required, opened a new log
I20240501 14:01:24.274276 2899 tablet_bootstrap.cc:492] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3: No bootstrap required, opened a new log
I20240501 14:01:24.274379 2898 ts_tablet_manager.cc:1367] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.013s
I20240501 14:01:24.274379 2900 ts_tablet_manager.cc:1367] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:01:24.274417 2899 ts_tablet_manager.cc:1367] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.004s
I20240501 14:01:24.275100 2899 raft_consensus.cc:348] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.275269 2899 raft_consensus.cc:374] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.275206 2898 raft_consensus.cc:348] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.275195 2900 raft_consensus.cc:348] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.275398 2898 raft_consensus.cc:374] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.275420 2900 raft_consensus.cc:374] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.275346 2899 raft_consensus.cc:729] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5ec466352142ce9f1b281c5e2784c3, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.275460 2898 raft_consensus.cc:729] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047859656efe4f979dbeb8e5bec3ad32, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.275766 2899 consensus_queue.cc:260] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.275866 2898 consensus_queue.cc:260] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.275476 2900 raft_consensus.cc:729] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a078c6056e74c4594b6fab2049c1b70, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.276894 2900 consensus_queue.cc:260] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.277421 2900 ts_tablet_manager.cc:1398] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:01:24.277676 2900 tablet_bootstrap.cc:492] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70: Bootstrap starting.
I20240501 14:01:24.279709 2900 tablet_bootstrap.cc:654] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.280560 2898 ts_tablet_manager.cc:1398] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20240501 14:01:24.281821 2898 tablet_bootstrap.cc:492] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32: Bootstrap starting.
I20240501 14:01:24.282048 2904 raft_consensus.cc:482] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:24.282218 2904 raft_consensus.cc:504] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.283255 2904 leader_election.cc:288] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969), 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
W20240501 14:01:24.297792 2730 tablet.cc:2369] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:01:24.303397 2898 tablet_bootstrap.cc:654] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.308012 2907 raft_consensus.cc:482] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:24.308223 2907 raft_consensus.cc:504] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.337031 2907 leader_election.cc:288] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969), 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193:39059)
I20240501 14:01:24.345312 2899 ts_tablet_manager.cc:1398] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3: Time spent starting tablet: real 0.071s user 0.043s sys 0.027s
I20240501 14:01:24.345752 2899 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3: Bootstrap starting.
I20240501 14:01:24.345799 2906 raft_consensus.cc:482] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:24.346014 2906 raft_consensus.cc:504] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.348007 2899 tablet_bootstrap.cc:654] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.356484 2899 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3: No bootstrap required, opened a new log
I20240501 14:01:24.356662 2898 tablet_bootstrap.cc:492] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32: No bootstrap required, opened a new log
I20240501 14:01:24.356664 2899 ts_tablet_manager.cc:1367] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 14:01:24.356902 2898 ts_tablet_manager.cc:1367] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.001s
I20240501 14:01:24.357729 2899 raft_consensus.cc:348] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.358012 2899 raft_consensus.cc:374] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.358070 2899 raft_consensus.cc:729] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5ec466352142ce9f1b281c5e2784c3, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.358381 2899 consensus_queue.cc:260] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.358983 2899 ts_tablet_manager.cc:1398] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:01:24.359318 2899 tablet_bootstrap.cc:492] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3: Bootstrap starting.
I20240501 14:01:24.361380 2899 tablet_bootstrap.cc:654] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.357862 2898 raft_consensus.cc:348] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.361877 2898 raft_consensus.cc:374] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.361958 2898 raft_consensus.cc:729] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047859656efe4f979dbeb8e5bec3ad32, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.362349 2898 consensus_queue.cc:260] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.362843 2898 ts_tablet_manager.cc:1398] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:01:24.363730 2898 tablet_bootstrap.cc:492] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32: Bootstrap starting.
I20240501 14:01:24.365746 2898 tablet_bootstrap.cc:654] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32: Neither blocks nor log segments found. Creating new log.
I20240501 14:01:24.381523 2906 leader_election.cc:288] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193:39059), 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
I20240501 14:01:24.389508 2899 tablet_bootstrap.cc:492] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3: No bootstrap required, opened a new log
I20240501 14:01:24.389700 2899 ts_tablet_manager.cc:1367] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.002s
I20240501 14:01:24.390714 2899 raft_consensus.cc:348] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.390980 2899 raft_consensus.cc:374] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.391031 2899 raft_consensus.cc:729] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5ec466352142ce9f1b281c5e2784c3, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.391320 2899 consensus_queue.cc:260] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.392302 2898 tablet_bootstrap.cc:492] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32: No bootstrap required, opened a new log
I20240501 14:01:24.392450 2898 ts_tablet_manager.cc:1367] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.016s
I20240501 14:01:24.394129 2899 ts_tablet_manager.cc:1398] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:01:24.395277 2898 raft_consensus.cc:348] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.395545 2898 raft_consensus.cc:374] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.395606 2898 raft_consensus.cc:729] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 047859656efe4f979dbeb8e5bec3ad32, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.395910 2898 consensus_queue.cc:260] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.396577 2898 ts_tablet_manager.cc:1398] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:01:24.400534 2900 tablet_bootstrap.cc:492] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70: No bootstrap required, opened a new log
I20240501 14:01:24.400676 2900 ts_tablet_manager.cc:1367] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70: Time spent bootstrapping tablet: real 0.123s user 0.040s sys 0.074s
I20240501 14:01:24.401576 2904 raft_consensus.cc:482] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:24.401484 2900 raft_consensus.cc:348] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.401680 2904 raft_consensus.cc:504] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.402576 2904 leader_election.cc:288] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181), cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969)
I20240501 14:01:24.401677 2900 raft_consensus.cc:374] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:01:24.402900 2900 raft_consensus.cc:729] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a078c6056e74c4594b6fab2049c1b70, State: Initialized, Role: FOLLOWER
I20240501 14:01:24.403151 2779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ffa91a5dd84bc8a7b09bd9a612595a" candidate_uuid: "047859656efe4f979dbeb8e5bec3ad32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ec466352142ce9f1b281c5e2784c3" is_pre_election: true
I20240501 14:01:24.403399 2779 raft_consensus.cc:2454] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 047859656efe4f979dbeb8e5bec3ad32 in term 0.
I20240501 14:01:24.403877 2900 consensus_queue.cc:260] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.405193 2819 leader_election.cc:302] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047859656efe4f979dbeb8e5bec3ad32, cc5ec466352142ce9f1b281c5e2784c3; no voters:
I20240501 14:01:24.405427 2912 raft_consensus.cc:2781] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:24.405521 2912 raft_consensus.cc:482] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:24.405611 2912 raft_consensus.cc:3037] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.405722 2900 ts_tablet_manager.cc:1398] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:01:24.408550 2912 raft_consensus.cc:504] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.409201 2912 leader_election.cc:288] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [CANDIDATE]: Term 1 election: Requested vote from peers cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969), 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193:39059)
I20240501 14:01:24.410034 2779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ffa91a5dd84bc8a7b09bd9a612595a" candidate_uuid: "047859656efe4f979dbeb8e5bec3ad32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ec466352142ce9f1b281c5e2784c3"
I20240501 14:01:24.410223 2779 raft_consensus.cc:3037] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.413100 2779 raft_consensus.cc:2454] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 047859656efe4f979dbeb8e5bec3ad32 in term 1.
I20240501 14:01:24.413475 2819 leader_election.cc:302] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047859656efe4f979dbeb8e5bec3ad32, cc5ec466352142ce9f1b281c5e2784c3; no voters:
I20240501 14:01:24.413786 2912 raft_consensus.cc:2781] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:24.413930 2912 raft_consensus.cc:686] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 1 LEADER]: Becoming Leader. State: Replica: 047859656efe4f979dbeb8e5bec3ad32, State: Running, Role: LEADER
I20240501 14:01:24.414191 2912 consensus_queue.cc:237] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } }
I20240501 14:01:24.414492 2853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302ae30a496f4ae79492fa6388e916ad" candidate_uuid: "cc5ec466352142ce9f1b281c5e2784c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047859656efe4f979dbeb8e5bec3ad32" is_pre_election: true
I20240501 14:01:24.414749 2853 raft_consensus.cc:2454] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc5ec466352142ce9f1b281c5e2784c3 in term 0.
I20240501 14:01:24.415675 2743 leader_election.cc:302] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047859656efe4f979dbeb8e5bec3ad32, cc5ec466352142ce9f1b281c5e2784c3; no voters:
I20240501 14:01:24.420559 2778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ffa91a5dd84bc8a7b09bd9a612595a" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ec466352142ce9f1b281c5e2784c3" is_pre_election: true
I20240501 14:01:24.420858 2778 raft_consensus.cc:2379] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4a078c6056e74c4594b6fab2049c1b70 in current term 1: Already voted for candidate 047859656efe4f979dbeb8e5bec3ad32 in this term.
I20240501 14:01:24.421432 2853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302ae30a496f4ae79492fa6388e916ad" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047859656efe4f979dbeb8e5bec3ad32" is_pre_election: true
I20240501 14:01:24.421523 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ffa91a5dd84bc8a7b09bd9a612595a" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047859656efe4f979dbeb8e5bec3ad32" is_pre_election: true
I20240501 14:01:24.421646 2853 raft_consensus.cc:2454] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a078c6056e74c4594b6fab2049c1b70 in term 0.
I20240501 14:01:24.420552 2779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302ae30a496f4ae79492fa6388e916ad" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ec466352142ce9f1b281c5e2784c3" is_pre_election: true
I20240501 14:01:24.422323 2779 raft_consensus.cc:2454] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a078c6056e74c4594b6fab2049c1b70 in term 0.
I20240501 14:01:24.422502 2668 leader_election.cc:302] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047859656efe4f979dbeb8e5bec3ad32, 4a078c6056e74c4594b6fab2049c1b70; no voters:
I20240501 14:01:24.422446 2602 catalog_manager.cc:5484] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195). New cstate: current_term: 1 leader_uuid: "047859656efe4f979dbeb8e5bec3ad32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:24.422739 2904 raft_consensus.cc:2781] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:24.422827 2904 raft_consensus.cc:482] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:24.422892 2904 raft_consensus.cc:3037] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.423305 2906 raft_consensus.cc:2781] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:24.424901 2906 raft_consensus.cc:482] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:24.425069 2906 raft_consensus.cc:3037] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.425418 2904 raft_consensus.cc:504] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.425995 2904 leader_election.cc:288] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 election: Requested vote from peers cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969), 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
I20240501 14:01:24.426349 2779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302ae30a496f4ae79492fa6388e916ad" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ec466352142ce9f1b281c5e2784c3"
I20240501 14:01:24.425477 2668 leader_election.cc:302] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a078c6056e74c4594b6fab2049c1b70; no voters: 047859656efe4f979dbeb8e5bec3ad32, cc5ec466352142ce9f1b281c5e2784c3
I20240501 14:01:24.427732 2904 raft_consensus.cc:3037] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.428166 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302ae30a496f4ae79492fa6388e916ad" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047859656efe4f979dbeb8e5bec3ad32"
I20240501 14:01:24.428318 2852 raft_consensus.cc:3037] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.428550 2906 raft_consensus.cc:504] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.428875 2779 raft_consensus.cc:2379] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a078c6056e74c4594b6fab2049c1b70 in current term 1: Already voted for candidate cc5ec466352142ce9f1b281c5e2784c3 in this term.
I20240501 14:01:24.429173 2906 leader_election.cc:288] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [CANDIDATE]: Term 1 election: Requested vote from peers 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193:39059), 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
I20240501 14:01:24.430377 2904 raft_consensus.cc:2726] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:01:24.430578 2704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302ae30a496f4ae79492fa6388e916ad" candidate_uuid: "cc5ec466352142ce9f1b281c5e2784c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a078c6056e74c4594b6fab2049c1b70" is_pre_election: true
I20240501 14:01:24.430704 2703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302ae30a496f4ae79492fa6388e916ad" candidate_uuid: "cc5ec466352142ce9f1b281c5e2784c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a078c6056e74c4594b6fab2049c1b70"
I20240501 14:01:24.430893 2704 raft_consensus.cc:2379] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cc5ec466352142ce9f1b281c5e2784c3 in current term 1: Already voted for candidate 4a078c6056e74c4594b6fab2049c1b70 in this term.
I20240501 14:01:24.430964 2852 raft_consensus.cc:2454] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a078c6056e74c4594b6fab2049c1b70 in term 1.
I20240501 14:01:24.431301 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302ae30a496f4ae79492fa6388e916ad" candidate_uuid: "cc5ec466352142ce9f1b281c5e2784c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047859656efe4f979dbeb8e5bec3ad32"
I20240501 14:01:24.431483 2852 raft_consensus.cc:2379] T 302ae30a496f4ae79492fa6388e916ad P 047859656efe4f979dbeb8e5bec3ad32 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cc5ec466352142ce9f1b281c5e2784c3 in current term 1: Already voted for candidate 4a078c6056e74c4594b6fab2049c1b70 in this term.
I20240501 14:01:24.431715 2704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ffa91a5dd84bc8a7b09bd9a612595a" candidate_uuid: "047859656efe4f979dbeb8e5bec3ad32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a078c6056e74c4594b6fab2049c1b70" is_pre_election: true
I20240501 14:01:24.431779 2668 leader_election.cc:302] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 047859656efe4f979dbeb8e5bec3ad32, 4a078c6056e74c4594b6fab2049c1b70; no voters: cc5ec466352142ce9f1b281c5e2784c3
I20240501 14:01:24.431869 2704 raft_consensus.cc:2454] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 047859656efe4f979dbeb8e5bec3ad32 in term 1.
I20240501 14:01:24.431843 2743 leader_election.cc:302] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cc5ec466352142ce9f1b281c5e2784c3; no voters: 047859656efe4f979dbeb8e5bec3ad32, 4a078c6056e74c4594b6fab2049c1b70
I20240501 14:01:24.432067 2703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6ffa91a5dd84bc8a7b09bd9a612595a" candidate_uuid: "047859656efe4f979dbeb8e5bec3ad32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a078c6056e74c4594b6fab2049c1b70"
I20240501 14:01:24.432195 2906 raft_consensus.cc:2726] T 302ae30a496f4ae79492fa6388e916ad P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:01:24.432550 2904 raft_consensus.cc:2781] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:24.434389 2703 raft_consensus.cc:2454] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 047859656efe4f979dbeb8e5bec3ad32 in term 1.
I20240501 14:01:24.434711 2904 raft_consensus.cc:686] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [term 1 LEADER]: Becoming Leader. State: Replica: 4a078c6056e74c4594b6fab2049c1b70, State: Running, Role: LEADER
I20240501 14:01:24.434950 2904 consensus_queue.cc:237] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.435566 2925 raft_consensus.cc:482] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:01:24.435667 2925 raft_consensus.cc:504] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.436615 2779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa36559e072427bafab53108b0d4dd2" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ec466352142ce9f1b281c5e2784c3" is_pre_election: true
I20240501 14:01:24.436663 2925 leader_election.cc:288] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969), 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
I20240501 14:01:24.436817 2779 raft_consensus.cc:2454] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a078c6056e74c4594b6fab2049c1b70 in term 0.
I20240501 14:01:24.437178 2670 leader_election.cc:302] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a078c6056e74c4594b6fab2049c1b70, cc5ec466352142ce9f1b281c5e2784c3; no voters:
I20240501 14:01:24.437391 2925 raft_consensus.cc:2781] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:01:24.437201 2600 catalog_manager.cc:5484] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193). New cstate: current_term: 1 leader_uuid: "4a078c6056e74c4594b6fab2049c1b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:24.437470 2925 raft_consensus.cc:482] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:01:24.437577 2925 raft_consensus.cc:3037] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.438458 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa36559e072427bafab53108b0d4dd2" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047859656efe4f979dbeb8e5bec3ad32" is_pre_election: true
I20240501 14:01:24.438628 2852 raft_consensus.cc:2454] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a078c6056e74c4594b6fab2049c1b70 in term 0.
I20240501 14:01:24.440073 2925 raft_consensus.cc:504] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.440728 2925 leader_election.cc:288] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 election: Requested vote from peers cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969), 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
I20240501 14:01:24.441020 2779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa36559e072427bafab53108b0d4dd2" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5ec466352142ce9f1b281c5e2784c3"
I20240501 14:01:24.441114 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa36559e072427bafab53108b0d4dd2" candidate_uuid: "4a078c6056e74c4594b6fab2049c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "047859656efe4f979dbeb8e5bec3ad32"
I20240501 14:01:24.441185 2779 raft_consensus.cc:3037] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.441253 2852 raft_consensus.cc:3037] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:01:24.442878 2779 raft_consensus.cc:2454] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a078c6056e74c4594b6fab2049c1b70 in term 1.
I20240501 14:01:24.442900 2852 raft_consensus.cc:2454] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a078c6056e74c4594b6fab2049c1b70 in term 1.
I20240501 14:01:24.443233 2668 leader_election.cc:302] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047859656efe4f979dbeb8e5bec3ad32, 4a078c6056e74c4594b6fab2049c1b70; no voters:
I20240501 14:01:24.443490 2925 raft_consensus.cc:2781] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:01:24.443588 2925 raft_consensus.cc:686] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 1 LEADER]: Becoming Leader. State: Replica: 4a078c6056e74c4594b6fab2049c1b70, State: Running, Role: LEADER
I20240501 14:01:24.443801 2925 consensus_queue.cc:237] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:24.445935 2600 catalog_manager.cc:5484] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193). New cstate: current_term: 1 leader_uuid: "4a078c6056e74c4594b6fab2049c1b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:24.450244 30475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:01:24.452919 2928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:24.453649 2929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:01:24.454355 2931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:01:24.454615 30475 server_base.cc:1034] running on GCE node
I20240501 14:01:24.454847 30475 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 14:01:24.454900 30475 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 14:01:24.454936 30475 hybrid_clock.cc:648] HybridClock initialized: now 1714572084454933 us; error 0 us; skew 500 ppm
I20240501 14:01:24.455092 30475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:01:24.456450 30475 webserver.cc:460] Webserver started at http://127.29.194.196:40975/ using document root <none> and password file <none>
I20240501 14:01:24.456657 30475 fs_manager.cc:359] Metadata directory not provided
I20240501 14:01:24.456768 30475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:01:24.456904 30475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:01:24.457675 30475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-3-root/instance:
uuid: "4295ea637f7c44ceb3dcae4293d660e2"
format_stamp: "Formatted at 2024-05-01 14:01:24 on dist-test-slave-kwk7"
I20240501 14:01:24.460287 30475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 14:01:24.461962 2936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:24.462342 30475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 14:01:24.462440 30475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-3-root
uuid: "4295ea637f7c44ceb3dcae4293d660e2"
format_stamp: "Formatted at 2024-05-01 14:01:24 on dist-test-slave-kwk7"
I20240501 14:01:24.462543 30475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskrA4pcO/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1714571949891039-30475-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:01:24.480975 30475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:01:24.481462 30475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:01:24.481958 30475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:01:24.482760 30475 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:01:24.482816 30475 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:24.482892 30475 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:01:24.482940 30475 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:01:24.492766 2912 consensus_queue.cc:1035] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:01:24.496213 2912 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20240501 14:01:24.496306 2912 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20240501 14:01:24.502895 30475 rpc_server.cc:307] RPC server started. Bound to: 127.29.194.196:42245
I20240501 14:01:24.502912 3000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.194.196:42245 every 8 connection(s)
I20240501 14:01:24.509124 3001 heartbeater.cc:347] Connected to a master server at 127.29.194.254:35515
I20240501 14:01:24.509251 3001 heartbeater.cc:464] Registering TS with master...
I20240501 14:01:24.509577 3001 heartbeater.cc:511] Master 127.29.194.254:35515 requested a full tablet report, sending...
I20240501 14:01:24.510280 2600 ts_manager.cc:194] Registered new tserver with Master: 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196:42245)
I20240501 14:01:24.510977 2600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54094
I20240501 14:01:24.523558 2907 consensus_queue.cc:1035] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:24.546188 2904 consensus_queue.cc:1035] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:24.547108 2904 consensus_queue.cc:1035] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:24.553232 2904 consensus_queue.cc:1035] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:24.560228 2904 consensus_queue.cc:1035] T 302ae30a496f4ae79492fa6388e916ad P 4a078c6056e74c4594b6fab2049c1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:01:24.968520 2655 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
W20240501 14:01:24.970034 2650 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table 3d7ded2460d84eafa21318b33ece48a1: could not find any suitable replica to move from server 4a078c6056e74c4594b6fab2049c1b70 to server 4295ea637f7c44ceb3dcae4293d660e2
I20240501 14:01:24.985977 2703 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "baa36559e072427bafab53108b0d4dd2"
dest_uuid: "4a078c6056e74c4594b6fab2049c1b70"
mode: GRACEFUL
new_leader_uuid: "cc5ec466352142ce9f1b281c5e2784c3"
from {username='slave'} at 127.0.0.1:39338
I20240501 14:01:24.986171 2703 raft_consensus.cc:595] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 1 LEADER]: Received request to transfer leadership to TS cc5ec466352142ce9f1b281c5e2784c3
I20240501 14:01:24.986562 2655 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 14:01:24.986683 2655 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 14:01:25.078200 2904 raft_consensus.cc:979] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70: : Instructing follower cc5ec466352142ce9f1b281c5e2784c3 to start an election
I20240501 14:01:25.078380 2904 raft_consensus.cc:1067] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 1 LEADER]: Signalling peer cc5ec466352142ce9f1b281c5e2784c3 to start an election
I20240501 14:01:25.078931 2779 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "baa36559e072427bafab53108b0d4dd2"
dest_uuid: "cc5ec466352142ce9f1b281c5e2784c3"
from {username='slave'} at 127.0.0.1:34624
I20240501 14:01:25.079103 2779 raft_consensus.cc:482] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:01:25.079212 2779 raft_consensus.cc:3037] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:25.082571 2779 raft_consensus.cc:504] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:25.083081 2779 leader_election.cc:288] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [CANDIDATE]: Term 2 election: Requested vote from peers 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193:39059), 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
I20240501 14:01:25.083451 2852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa36559e072427bafab53108b0d4dd2" candidate_uuid: "cc5ec466352142ce9f1b281c5e2784c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "047859656efe4f979dbeb8e5bec3ad32"
I20240501 14:01:25.083607 2852 raft_consensus.cc:3037] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:25.083739 2703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa36559e072427bafab53108b0d4dd2" candidate_uuid: "cc5ec466352142ce9f1b281c5e2784c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a078c6056e74c4594b6fab2049c1b70"
I20240501 14:01:25.083897 2703 raft_consensus.cc:3032] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:01:25.083966 2703 raft_consensus.cc:729] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a078c6056e74c4594b6fab2049c1b70, State: Running, Role: LEADER
I20240501 14:01:25.084193 2703 consensus_queue.cc:260] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:25.084484 2703 raft_consensus.cc:3037] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:01:25.085709 2852 raft_consensus.cc:2454] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc5ec466352142ce9f1b281c5e2784c3 in term 2.
I20240501 14:01:25.086035 2743 leader_election.cc:302] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 047859656efe4f979dbeb8e5bec3ad32, cc5ec466352142ce9f1b281c5e2784c3; no voters:
I20240501 14:01:25.086606 2703 raft_consensus.cc:2454] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc5ec466352142ce9f1b281c5e2784c3 in term 2.
I20240501 14:01:25.086827 3015 raft_consensus.cc:2781] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:01:25.087188 3015 raft_consensus.cc:686] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 2 LEADER]: Becoming Leader. State: Replica: cc5ec466352142ce9f1b281c5e2784c3, State: Running, Role: LEADER
I20240501 14:01:25.087416 3015 consensus_queue.cc:237] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } }
I20240501 14:01:25.089306 2600 catalog_manager.cc:5484] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 reported cstate change: term changed from 1 to 2, leader changed from 4a078c6056e74c4594b6fab2049c1b70 (127.29.194.193) to cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194). New cstate: current_term: 2 leader_uuid: "cc5ec466352142ce9f1b281c5e2784c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } health_report { overall_health: UNKNOWN } } }
I20240501 14:01:25.192466 2852 raft_consensus.cc:1261] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 2 FOLLOWER]: Refusing update from remote peer cc5ec466352142ce9f1b281c5e2784c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:25.192965 3015 consensus_queue.cc:1035] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:25.193277 2703 raft_consensus.cc:1261] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 2 FOLLOWER]: Refusing update from remote peer cc5ec466352142ce9f1b281c5e2784c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:01:25.193672 3015 consensus_queue.cc:1035] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:25.512039 3001 heartbeater.cc:503] Master 127.29.194.254:35515 was elected leader, sending a full tablet report...
I20240501 14:01:25.978214 2852 consensus_queue.cc:237] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } attrs { replace: true } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } }
I20240501 14:01:25.980424 2703 raft_consensus.cc:1261] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Refusing update from remote peer 047859656efe4f979dbeb8e5bec3ad32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:25.980424 2779 raft_consensus.cc:1261] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Refusing update from remote peer 047859656efe4f979dbeb8e5bec3ad32: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:01:25.981427 3028 consensus_queue.cc:1035] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:25.981734 3027 consensus_queue.cc:1035] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 14:01:25.984994 2779 raft_consensus.cc:2932] T a6ffa91a5dd84bc8a7b09bd9a612595a P cc5ec466352142ce9f1b281c5e2784c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } attrs { replace: true } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } } }
I20240501 14:01:25.985517 2703 raft_consensus.cc:2932] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } attrs { replace: true } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } } }
W20240501 14:01:25.987977 2816 consensus_peers.cc:485] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 -> Peer 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196:42245): Couldn't send request to peer 4295ea637f7c44ceb3dcae4293d660e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a6ffa91a5dd84bc8a7b09bd9a612595a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:25.990113 2600 catalog_manager.cc:5484] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4a078c6056e74c4594b6fab2049c1b70 reported cstate change: config changed from index -1 to 2, NON_VOTER 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196) added. New cstate: current_term: 1 leader_uuid: "047859656efe4f979dbeb8e5bec3ad32" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } attrs { replace: true } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } } }
I20240501 14:01:26.036552 3028 raft_consensus.cc:2932] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } attrs { replace: true } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } } }
I20240501 14:01:26.046401 2779 consensus_queue.cc:237] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } attrs { replace: true } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } }
I20240501 14:01:26.048287 2703 raft_consensus.cc:1261] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 2 FOLLOWER]: Refusing update from remote peer cc5ec466352142ce9f1b281c5e2784c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:26.048949 2852 raft_consensus.cc:1261] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 2 FOLLOWER]: Refusing update from remote peer cc5ec466352142ce9f1b281c5e2784c3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:01:26.049005 3036 consensus_queue.cc:1035] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:26.049489 3016 consensus_queue.cc:1035] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:01:26.052016 2703 raft_consensus.cc:2932] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } attrs { replace: true } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } } }
W20240501 14:01:26.058765 2742 consensus_peers.cc:485] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 -> Peer 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196:42245): Couldn't send request to peer 4295ea637f7c44ceb3dcae4293d660e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: baa36559e072427bafab53108b0d4dd2. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:01:26.059250 2852 raft_consensus.cc:2932] T baa36559e072427bafab53108b0d4dd2 P 047859656efe4f979dbeb8e5bec3ad32 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } attrs { replace: true } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } } }
I20240501 14:01:26.061743 2602 catalog_manager.cc:5484] T baa36559e072427bafab53108b0d4dd2 P 4a078c6056e74c4594b6fab2049c1b70 reported cstate change: config changed from index -1 to 3, NON_VOTER 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196) added. New cstate: current_term: 2 leader_uuid: "cc5ec466352142ce9f1b281c5e2784c3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } attrs { replace: true } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } } }
I20240501 14:01:26.097896 3040 ts_tablet_manager.cc:902] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: Initiating tablet copy from peer 047859656efe4f979dbeb8e5bec3ad32 (127.29.194.195:39181)
I20240501 14:01:26.099411 3040 tablet_copy_client.cc:308] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: tablet copy: Beginning tablet copy session from remote peer at address 127.29.194.195:39181
I20240501 14:01:26.104939 2863 tablet_copy_service.cc:140] P 047859656efe4f979dbeb8e5bec3ad32: Received BeginTabletCopySession request for tablet a6ffa91a5dd84bc8a7b09bd9a612595a from peer 4295ea637f7c44ceb3dcae4293d660e2 ({username='slave'} at 127.0.0.1:33278)
I20240501 14:01:26.105062 2863 tablet_copy_service.cc:161] P 047859656efe4f979dbeb8e5bec3ad32: Beginning new tablet copy session on tablet a6ffa91a5dd84bc8a7b09bd9a612595a from peer 4295ea637f7c44ceb3dcae4293d660e2 at {username='slave'} at 127.0.0.1:33278: session id = 4295ea637f7c44ceb3dcae4293d660e2-a6ffa91a5dd84bc8a7b09bd9a612595a
I20240501 14:01:26.106967 2863 tablet_copy_source_session.cc:205] T a6ffa91a5dd84bc8a7b09bd9a612595a P 047859656efe4f979dbeb8e5bec3ad32: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:01:26.108023 3040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6ffa91a5dd84bc8a7b09bd9a612595a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:01:26.112319 3040 tablet_copy_client.cc:783] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: tablet copy: Starting download of 0 data blocks...
I20240501 14:01:26.112634 3040 tablet_copy_client.cc:647] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: tablet copy: Starting download of 1 WAL segments...
I20240501 14:01:26.113989 3040 tablet_copy_client.cc:523] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:01:26.116325 3040 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: Bootstrap starting.
I20240501 14:01:26.120936 3040 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:01:26.121241 3040 tablet_bootstrap.cc:492] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: Bootstrap complete.
I20240501 14:01:26.121470 3040 ts_tablet_manager.cc:1367] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.001s
I20240501 14:01:26.122195 3040 raft_consensus.cc:348] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } attrs { replace: true } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } }
I20240501 14:01:26.122380 3040 raft_consensus.cc:729] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4295ea637f7c44ceb3dcae4293d660e2, State: Initialized, Role: LEARNER
I20240501 14:01:26.122658 3040 consensus_queue.cc:260] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } attrs { replace: true } } peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } }
I20240501 14:01:26.123442 3040 ts_tablet_manager.cc:1398] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:01:26.124130 2863 tablet_copy_service.cc:342] P 047859656efe4f979dbeb8e5bec3ad32: Request end of tablet copy session 4295ea637f7c44ceb3dcae4293d660e2-a6ffa91a5dd84bc8a7b09bd9a612595a received from {username='slave'} at 127.0.0.1:33278
I20240501 14:01:26.124287 2863 tablet_copy_service.cc:433] P 047859656efe4f979dbeb8e5bec3ad32: ending tablet copy session 4295ea637f7c44ceb3dcae4293d660e2-a6ffa91a5dd84bc8a7b09bd9a612595a on tablet a6ffa91a5dd84bc8a7b09bd9a612595a with peer 4295ea637f7c44ceb3dcae4293d660e2
I20240501 14:01:26.124647 3016 raft_consensus.cc:2932] T baa36559e072427bafab53108b0d4dd2 P cc5ec466352142ce9f1b281c5e2784c3 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4295ea637f7c44ceb3dcae4293d660e2 (127.29.194.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4a078c6056e74c4594b6fab2049c1b70" member_type: VOTER last_known_addr { host: "127.29.194.193" port: 39059 } attrs { replace: true } } peers { permanent_uuid: "cc5ec466352142ce9f1b281c5e2784c3" member_type: VOTER last_known_addr { host: "127.29.194.194" port: 41969 } } peers { permanent_uuid: "047859656efe4f979dbeb8e5bec3ad32" member_type: VOTER last_known_addr { host: "127.29.194.195" port: 39181 } } peers { permanent_uuid: "4295ea637f7c44ceb3dcae4293d660e2" member_type: NON_VOTER last_known_addr { host: "127.29.194.196" port: 42245 } attrs { promote: true } } }
I20240501 14:01:26.128177 3040 ts_tablet_manager.cc:902] T baa36559e072427bafab53108b0d4dd2 P 4295ea637f7c44ceb3dcae4293d660e2: Initiating tablet copy from peer cc5ec466352142ce9f1b281c5e2784c3 (127.29.194.194:41969)
F20240501 14:01:26.128531 3040 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2bdda4e923dc4caaba534f1e8b5693e2.queue_time_us, tablet-download-pool-2bdda4e923dc4caaba534f1e8b5693e2.queue_time_us) (tablet-download-pool-2bdda4e923dc4caaba534f1e8b5693e2.run_cpu_time_us, tablet-download-pool-2bdda4e923dc4caaba534f1e8b5693e2.run_cpu_time_us) (tablet-download-pool-2bdda4e923dc4caaba534f1e8b5693e2.run_wall_time_us, tablet-download-pool-2bdda4e923dc4caaba534f1e8b5693e2.run_wall_time_us) (tablet-download-pool-348221e7dbd04efcba6185ba29a33a60.queue_time_us, tablet-download-pool-348221e7dbd04efcba6185ba29a33a60.queue_time_us) (tablet-download-pool-348221e7dbd04efcba6185ba29a33a60.run_cpu_time_us, tablet-download-pool-348221e7dbd04efcba6185ba29a33a60.run_cpu_time_us) (tablet-download-pool-348221e7dbd04efcba6185ba29a33a60.run_wall_time_us, tablet-download-pool-348221e7dbd04efcba6185ba29a33a60.run_wall_time_us) (tablet-download-pool-355ae59edd1c4620904ba23275021582.queue_time_us, tablet-download-pool-355ae59edd1c4620904ba23275021582.queue_time_us) (tablet-download-pool-355ae59edd1c4620904ba23275021582.run_cpu_time_us, tablet-download-pool-355ae59edd1c4620904ba23275021582.run_cpu_time_us) (tablet-download-pool-355ae59edd1c4620904ba23275021582.run_wall_time_us, tablet-download-pool-355ae59edd1c4620904ba23275021582.run_wall_time_us) (tablet-download-pool-46ae855dc1a44159b5f3cca0657dc752.queue_time_us, tablet-download-pool-46ae855dc1a44159b5f3cca0657dc752.queue_time_us) (tablet-download-pool-46ae855dc1a44159b5f3cca0657dc752.run_cpu_time_us, tablet-download-pool-46ae855dc1a44159b5f3cca0657dc752.run_cpu_time_us) (tablet-download-pool-46ae855dc1a44159b5f3cca0657dc752.run_wall_time_us, tablet-download-pool-46ae855dc1a44159b5f3cca0657dc752.run_wall_time_us) (tablet-download-pool-6ee8e1d3ee3b429ebd15a32afe393e69.queue_time_us, tablet-download-pool-6ee8e1d3ee3b429ebd15a32afe393e69.queue_time_us) (tablet-download-pool-6ee8e1d3ee3b429ebd15a32afe393e69.run_cpu_time_us, tablet-download-pool-6ee8e1d3ee3b429ebd15a32afe393e69.run_cpu_time_us) (tablet-download-pool-6ee8e1d3ee3b429ebd15a32afe393e69.run_wall_time_us, tablet-download-pool-6ee8e1d3ee3b429ebd15a32afe393e69.run_wall_time_us) (tablet-download-pool-78e839ad73564640b24fe9efc9f41f08.queue_time_us, tablet-download-pool-78e839ad73564640b24fe9efc9f41f08.queue_time_us) (tablet-download-pool-78e839ad73564640b24fe9efc9f41f08.run_cpu_time_us, tablet-download-pool-78e839ad73564640b24fe9efc9f41f08.run_cpu_time_us) (tablet-download-pool-78e839ad73564640b24fe9efc9f41f08.run_wall_time_us, tablet-download-pool-78e839ad73564640b24fe9efc9f41f08.run_wall_time_us) (tablet-download-pool-7ca80506974245cb887559f7f884fb98.queue_time_us, tablet-download-pool-7ca80506974245cb887559f7f884fb98.queue_time_us) (tablet-download-pool-7ca80506974245cb887559f7f884fb98.run_cpu_time_us, tablet-download-pool-7ca80506974245cb887559f7f884fb98.run_cpu_time_us) (tablet-download-pool-7ca80506974245cb887559f7f884fb98.run_wall_time_us, tablet-download-pool-7ca80506974245cb887559f7f884fb98.run_wall_time_us) (tablet-download-pool-8c02ea2c5d074862a9286ec08fcb2cb4.queue_time_us, tablet-download-pool-8c02ea2c5d074862a9286ec08fcb2cb4.queue_time_us) (tablet-download-pool-8c02ea2c5d074862a9286ec08fcb2cb4.run_cpu_time_us, tablet-download-pool-8c02ea2c5d074862a9286ec08fcb2cb4.run_cpu_time_us) (tablet-download-pool-8c02ea2c5d074862a9286ec08fcb2cb4.run_wall_time_us, tablet-download-pool-8c02ea2c5d074862a9286ec08fcb2cb4.run_wall_time_us) (tablet-download-pool-8c36a1a8c09d415d8c1d1860c0d8b622.queue_time_us, tablet-download-pool-8c36a1a8c09d415d8c1d1860c0d8b622.queue_time_us) (tablet-download-pool-8c36a1a8c09d415d8c1d1860c0d8b622.run_cpu_time_us, tablet-download-pool-8c36a1a8c09d415d8c1d1860c0d8b622.run_cpu_time_us) (tablet-download-pool-8c36a1a8c09d415d8c1d1860c0d8b622.run_wall_time_us, tablet-download-pool-8c36a1a8c09d415d8c1d1860c0d8b622.run_wall_time_us) (tablet-download-pool-916a3f31691044f783d655695bf0a369.queue_time_us, tablet-download-pool-916a3f31691044f783d655695bf0a369.queue_time_us) (tablet-download-pool-916a3f31691044f783d655695bf0a369.run_cpu_time_us, tablet-download-pool-916a3f31691044f783d655695bf0a369.run_cpu_time_us) (tablet-download-pool-916a3f31691044f783d655695bf0a369.run_wall_time_us, tablet-download-pool-916a3f31691044f783d655695bf0a369.run_wall_time_us) (tablet-download-pool-a6ffa91a5dd84bc8a7b09bd9a612595a.queue_time_us, tablet-download-pool-a6ffa91a5dd84bc8a7b09bd9a612595a.queue_time_us) (tablet-download-pool-a6ffa91a5dd84bc8a7b09bd9a612595a.run_cpu_time_us, tablet-download-pool-a6ffa91a5dd84bc8a7b09bd9a612595a.run_cpu_time_us) (tablet-download-pool-a6ffa91a5dd84bc8a7b09bd9a612595a.run_wall_time_us, tablet-download-pool-a6ffa91a5dd84bc8a7b09bd9a612595a.run_wall_time_us) (tablet-download-pool-ae2d33db85414876998a3ed550e44e41.queue_time_us, tablet-download-pool-ae2d33db85414876998a3ed550e44e41.queue_time_us) (tablet-download-pool-ae2d33db85414876998a3ed550e44e41.run_cpu_time_us, tablet-download-pool-ae2d33db85414876998a3ed550e44e41.run_cpu_time_us) (tablet-download-pool-ae2d33db85414876998a3ed550e44e41.run_wall_time_us, tablet-download-pool-ae2d33db85414876998a3ed550e44e41.run_wall_time_us) (tablet-download-pool-baa36559e072427bafab53108b0d4dd2.queue_time_us, tablet-download-pool-baa36559e072427bafab53108b0d4dd2.queue_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714572086 (unix time) try "date -d @1714572086" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000770b) received by PID 30475 (TID 0x7f28691ab700) from PID 30475; stack trace: ***
@ 0x7f290f350980 (unknown) at ??:0
@ 0x7f290469dfb7 gsignal at ??:0
@ 0x7f290469f921 abort at ??:0
@ 0x7f2907152dcd google::LogMessage::Fail() at ??:0
@ 0x7f2907156b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f29071527cc google::LogMessage::Flush() at ??:0
@ 0x7f2907153f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f29087cf742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f290879bf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f2908795cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f2917c618de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f2917c79daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:01:26.192488 2974 raft_consensus.cc:1203] T a6ffa91a5dd84bc8a7b09bd9a612595a P 4295ea637f7c44ceb3dcae4293d660e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7f2917da4480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f2917ddca8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f2917dbf981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f2917dbf70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f291cc6198d std::function<>::operator()() at ??:0
@ 0x7f290879fb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f29087a319c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f29087a3019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f291cc6198d std::function<>::operator()() at ??:0
@ 0x7f290877e01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f290f3456db start_thread at ??:0
@ 0x7f290478071f clone at ??:0