[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 13:59:45.277961 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:33115,127.0.99.61:46569,127.0.99.60:40511
I20240503 13:59:45.279120 396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 13:59:45.279736 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:45.288165 401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:45.288792 404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:45.289382 402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.289932 396 server_base.cc:1034] running on GCE node
I20240503 13:59:45.290534 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:45.290660 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:45.290704 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744785290701 us; error 0 us; skew 500 ppm
I20240503 13:59:45.298393 396 webserver.cc:460] Webserver started at http://127.0.99.62:43839/ using document root <none> and password file <none>
I20240503 13:59:45.299062 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:45.299132 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:45.299371 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:45.301345 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "aaf89f2a0fdd42c3882ef57f112b215e"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.305604 396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240503 13:59:45.308317 409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.309682 396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:45.309823 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "aaf89f2a0fdd42c3882ef57f112b215e"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.309935 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:45.324967 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:45.325716 396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 13:59:45.325909 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:45.333320 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:33115
I20240503 13:59:45.333359 461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:33115 every 8 connection(s)
I20240503 13:59:45.334658 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:45.337169 462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 13:59:45.337270 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:45.337792 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:45.338275 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.339154 396 server_base.cc:1034] running on GCE node
I20240503 13:59:45.339367 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:45.339417 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:45.339447 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744785339446 us; error 0 us; skew 500 ppm
I20240503 13:59:45.340560 396 webserver.cc:460] Webserver started at http://127.0.99.61:37053/ using document root <none> and password file <none>
I20240503 13:59:45.340730 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:45.340788 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:45.340864 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:45.341293 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root/instance:
uuid: "d84bcb85a3304a19b7e3f866c0ba5f30"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.342962 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:45.343343 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:45.344444 472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.344749 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.344818 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root
uuid: "d84bcb85a3304a19b7e3f866c0ba5f30"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.344890 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:45.355237 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } has no permanent_uuid. Determining permanent_uuid...
W20240503 13:59:45.356127 412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.99.61:46569: connect: Connection refused (error 111)
W20240503 13:59:45.356986 462 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.61:46569: Network error: Client connection negotiation failed: client connection to 127.0.99.61:46569: connect: Connection refused (error 111)
I20240503 13:59:45.358414 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:45.358793 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:45.363368 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.61:46569
I20240503 13:59:45.363409 527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.61:46569 every 8 connection(s)
I20240503 13:59:45.364149 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:45.364341 528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.365674 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } has no permanent_uuid. Determining permanent_uuid...
W20240503 13:59:45.366807 530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:45.367866 531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:45.368687 533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.369364 396 server_base.cc:1034] running on GCE node
I20240503 13:59:45.369537 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:45.369576 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:45.369590 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744785369590 us; error 0 us; skew 500 ppm
I20240503 13:59:45.370601 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:45.370784 396 webserver.cc:460] Webserver started at http://127.0.99.60:37001/ using document root <none> and password file <none>
I20240503 13:59:45.370934 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:45.370982 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:45.371099 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:45.371533 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root/instance:
uuid: "453b677f710d43a39288ff37d4f050aa"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.373108 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.374275 540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.374527 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.374575 528 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:45.374601 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root
uuid: "453b677f710d43a39288ff37d4f050aa"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.374681 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240503 13:59:45.375918 528 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:40511: Network error: Client connection negotiation failed: client connection to 127.0.99.60:40511: connect: Connection refused (error 111)
I20240503 13:59:45.388809 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:45.389278 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:45.394137 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.60:40511
I20240503 13:59:45.394304 593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.60:40511 every 8 connection(s)
I20240503 13:59:45.394615 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 13:59:45.395093 594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:45.396397 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:45.400749 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:45.405020 594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:45.406179 462 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } attempt: 1
I20240503 13:59:45.410104 462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:45.411682 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa: Bootstrap starting.
I20240503 13:59:45.414750 462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: Bootstrap starting.
I20240503 13:59:45.414757 594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.415620 462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.415753 594 log.cc:824] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa: Log is configured to *not* fsync() on all Append() calls
I20240503 13:59:45.417371 462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: No bootstrap required, opened a new log
I20240503 13:59:45.417371 594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa: No bootstrap required, opened a new log
I20240503 13:59:45.420905 594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.420919 462 raft_consensus.cc:348] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.421125 594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.421134 462 raft_consensus.cc:374] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.421205 462 raft_consensus.cc:729] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaf89f2a0fdd42c3882ef57f112b215e, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.421207 594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453b677f710d43a39288ff37d4f050aa, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.421675 462 consensus_queue.cc:260] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.421675 594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.422179 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.422307 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:45.422487 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.422566 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:45.422959 462 sys_catalog.cc:564] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:45.422960 594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:45.425096 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 13:59:45.428073 528 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } attempt: 1
W20240503 13:59:45.428493 623 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 13:59:45.428573 623 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 13:59:45.429004 622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 13:59:45.429075 622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 13:59:45.433470 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30: Bootstrap starting.
I20240503 13:59:45.434684 528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:45.436106 528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30: No bootstrap required, opened a new log
I20240503 13:59:45.436647 528 raft_consensus.cc:348] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.436801 528 raft_consensus.cc:374] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:45.436836 528 raft_consensus.cc:729] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d84bcb85a3304a19b7e3f866c0ba5f30, State: Initialized, Role: FOLLOWER
I20240503 13:59:45.436995 528 consensus_queue.cc:260] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.437379 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.437489 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:45.437574 528 sys_catalog.cc:564] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:45.439807 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 13:59:45.440559 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:45.440553 637 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 13:59:45.440654 637 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 13:59:45.443506 638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:45.443966 639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.445659 396 server_base.cc:1034] running on GCE node
W20240503 13:59:45.445950 641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.446235 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:45.446305 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:45.446326 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744785446326 us; error 0 us; skew 500 ppm
I20240503 13:59:45.447526 396 webserver.cc:460] Webserver started at http://127.0.99.1:39057/ using document root <none> and password file <none>
I20240503 13:59:45.447708 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:45.447783 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:45.447871 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:45.448316 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root/instance:
uuid: "56d8cff9496741b7b1c324a1a61ac531"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.449940 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:45.450965 646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.451269 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.451340 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root
uuid: "56d8cff9496741b7b1c324a1a61ac531"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.451417 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:45.472620 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:45.473229 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:45.473759 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:45.474650 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:45.474716 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.474771 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:45.474793 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.481005 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:45993
I20240503 13:59:45.481029 709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:45993 every 8 connection(s)
I20240503 13:59:45.482239 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:45.488756 711 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33115
I20240503 13:59:45.488924 711 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.489027 712 heartbeater.cc:347] Connected to a master server at 127.0.99.61:46569
I20240503 13:59:45.489125 712 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.489233 396 server_base.cc:1034] running on GCE node
I20240503 13:59:45.489634 712 heartbeater.cc:511] Master 127.0.99.61:46569 requested a full tablet report, sending...
I20240503 13:59:45.488947 710 heartbeater.cc:347] Connected to a master server at 127.0.99.60:40511
I20240503 13:59:45.489917 710 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.490113 710 heartbeater.cc:511] Master 127.0.99.60:40511 requested a full tablet report, sending...
W20240503 13:59:45.490273 716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.491218 492 ts_manager.cc:194] Registered new tserver with Master: 56d8cff9496741b7b1c324a1a61ac531 (127.0.99.1:45993)
I20240503 13:59:45.489634 711 heartbeater.cc:511] Master 127.0.99.62:33115 requested a full tablet report, sending...
I20240503 13:59:45.491231 558 ts_manager.cc:194] Registered new tserver with Master: 56d8cff9496741b7b1c324a1a61ac531 (127.0.99.1:45993)
W20240503 13:59:45.491744 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.492063 426 ts_manager.cc:194] Registered new tserver with Master: 56d8cff9496741b7b1c324a1a61ac531 (127.0.99.1:45993)
W20240503 13:59:45.492244 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.492477 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:45.492522 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:45.492542 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744785492540 us; error 0 us; skew 500 ppm
I20240503 13:59:45.493657 396 webserver.cc:460] Webserver started at http://127.0.99.2:45791/ using document root <none> and password file <none>
I20240503 13:59:45.493809 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:45.493856 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:45.493932 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:45.494386 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root/instance:
uuid: "be42e397639b44c99537496653abc38c"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.496114 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:45.497251 725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.497570 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.497639 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root
uuid: "be42e397639b44c99537496653abc38c"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.497720 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:45.508013 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:45.508841 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:45.509176 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:45.509701 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:45.509747 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.509781 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:45.509809 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.515997 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:33109
I20240503 13:59:45.516044 788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:33109 every 8 connection(s)
I20240503 13:59:45.517123 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:45.520721 796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:45.521430 795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.521731 396 server_base.cc:1034] running on GCE node
I20240503 13:59:45.522518 789 heartbeater.cc:347] Connected to a master server at 127.0.99.60:40511
I20240503 13:59:45.522624 789 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.522854 789 heartbeater.cc:511] Master 127.0.99.60:40511 requested a full tablet report, sending...
W20240503 13:59:45.523134 798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:45.523415 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:45.523463 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:45.523484 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744785523483 us; error 0 us; skew 500 ppm
I20240503 13:59:45.523777 558 ts_manager.cc:194] Registered new tserver with Master: be42e397639b44c99537496653abc38c (127.0.99.2:33109)
I20240503 13:59:45.523932 790 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33115
I20240503 13:59:45.524006 790 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.524291 790 heartbeater.cc:511] Master 127.0.99.62:33115 requested a full tablet report, sending...
I20240503 13:59:45.525053 426 ts_manager.cc:194] Registered new tserver with Master: be42e397639b44c99537496653abc38c (127.0.99.2:33109)
I20240503 13:59:45.525243 396 webserver.cc:460] Webserver started at http://127.0.99.3:46569/ using document root <none> and password file <none>
I20240503 13:59:45.525417 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:45.525458 791 heartbeater.cc:347] Connected to a master server at 127.0.99.61:46569
I20240503 13:59:45.525480 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:45.525539 791 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.525575 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:45.525751 791 heartbeater.cc:511] Master 127.0.99.61:46569 requested a full tablet report, sending...
I20240503 13:59:45.526034 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root/instance:
uuid: "455e7a39f4b1470dad396c6315d191b7"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.526315 492 ts_manager.cc:194] Registered new tserver with Master: be42e397639b44c99537496653abc38c (127.0.99.2:33109)
I20240503 13:59:45.527827 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20240503 13:59:45.528795 804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.529094 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:45.529162 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root
uuid: "455e7a39f4b1470dad396c6315d191b7"
format_stamp: "Formatted at 2024-05-03 13:59:45 on dist-test-slave-6w8v"
I20240503 13:59:45.529228 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:45.538229 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:45.538770 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:45.539113 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:45.539713 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:45.539762 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.539795 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:45.539822 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:45.546991 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:40249
I20240503 13:59:45.547135 867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:40249 every 8 connection(s)
I20240503 13:59:45.553606 869 heartbeater.cc:347] Connected to a master server at 127.0.99.62:33115
I20240503 13:59:45.553784 869 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.553977 870 heartbeater.cc:347] Connected to a master server at 127.0.99.61:46569
I20240503 13:59:45.554050 870 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.554088 868 heartbeater.cc:347] Connected to a master server at 127.0.99.60:40511
I20240503 13:59:45.554193 868 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:45.554200 870 heartbeater.cc:511] Master 127.0.99.61:46569 requested a full tablet report, sending...
I20240503 13:59:45.554546 868 heartbeater.cc:511] Master 127.0.99.60:40511 requested a full tablet report, sending...
I20240503 13:59:45.554114 869 heartbeater.cc:511] Master 127.0.99.62:33115 requested a full tablet report, sending...
I20240503 13:59:45.555025 492 ts_manager.cc:194] Registered new tserver with Master: 455e7a39f4b1470dad396c6315d191b7 (127.0.99.3:40249)
I20240503 13:59:45.555271 558 ts_manager.cc:194] Registered new tserver with Master: 455e7a39f4b1470dad396c6315d191b7 (127.0.99.3:40249)
I20240503 13:59:45.555681 426 ts_manager.cc:194] Registered new tserver with Master: 455e7a39f4b1470dad396c6315d191b7 (127.0.99.3:40249)
I20240503 13:59:45.556597 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008849921s
W20240503 13:59:45.577162 415 tablet.cc:2369] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 13:59:45.597168 600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:45.597426 600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.598434 600 leader_election.cc:288] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaf89f2a0fdd42c3882ef57f112b215e (127.0.99.62:33115), d84bcb85a3304a19b7e3f866c0ba5f30 (127.0.99.61:46569)
I20240503 13:59:45.599113 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "453b677f710d43a39288ff37d4f050aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" is_pre_election: true
I20240503 13:59:45.599126 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "453b677f710d43a39288ff37d4f050aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" is_pre_election: true
I20240503 13:59:45.599432 436 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453b677f710d43a39288ff37d4f050aa in term 0.
I20240503 13:59:45.599432 502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 453b677f710d43a39288ff37d4f050aa in term 0.
I20240503 13:59:45.599927 544 leader_election.cc:302] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453b677f710d43a39288ff37d4f050aa, d84bcb85a3304a19b7e3f866c0ba5f30; no voters:
I20240503 13:59:45.600209 600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:45.600286 600 raft_consensus.cc:482] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:45.600340 600 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.601549 600 raft_consensus.cc:504] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.601984 600 leader_election.cc:288] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [CANDIDATE]: Term 1 election: Requested vote from peers aaf89f2a0fdd42c3882ef57f112b215e (127.0.99.62:33115), d84bcb85a3304a19b7e3f866c0ba5f30 (127.0.99.61:46569)
I20240503 13:59:45.602391 436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "453b677f710d43a39288ff37d4f050aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaf89f2a0fdd42c3882ef57f112b215e"
I20240503 13:59:45.602515 502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "453b677f710d43a39288ff37d4f050aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30"
I20240503 13:59:45.602599 436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.602612 502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:45.603482 436 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453b677f710d43a39288ff37d4f050aa in term 1.
I20240503 13:59:45.603698 502 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 453b677f710d43a39288ff37d4f050aa in term 1.
I20240503 13:59:45.603830 545 leader_election.cc:302] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 453b677f710d43a39288ff37d4f050aa, aaf89f2a0fdd42c3882ef57f112b215e; no voters:
I20240503 13:59:45.604029 600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:45.604264 600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 1 LEADER]: Becoming Leader. State: Replica: 453b677f710d43a39288ff37d4f050aa, State: Running, Role: LEADER
I20240503 13:59:45.604610 600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } }
I20240503 13:59:45.606799 876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453b677f710d43a39288ff37d4f050aa. Latest consensus state: current_term: 1 leader_uuid: "453b677f710d43a39288ff37d4f050aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.606949 876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:45.607400 878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:45.608333 878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:45.614040 436 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 1 FOLLOWER]: Refusing update from remote peer 453b677f710d43a39288ff37d4f050aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:45.614040 502 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 1 FOLLOWER]: Refusing update from remote peer 453b677f710d43a39288ff37d4f050aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:45.614735 600 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.614934 876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:45.616955 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453b677f710d43a39288ff37d4f050aa. Latest consensus state: current_term: 1 leader_uuid: "453b677f710d43a39288ff37d4f050aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.617097 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:45.617089 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453b677f710d43a39288ff37d4f050aa. Latest consensus state: current_term: 1 leader_uuid: "453b677f710d43a39288ff37d4f050aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.617214 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:45.618902 600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "453b677f710d43a39288ff37d4f050aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.618956 876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "453b677f710d43a39288ff37d4f050aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.619019 600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:45.619071 876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:45.619135 601 sys_catalog.cc:455] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "453b677f710d43a39288ff37d4f050aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.619203 601 sys_catalog.cc:458] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:45.619237 881 mvcc.cc:205] Tried to move back new op lower bound from 7023594641867259904 to 7023594641839816704. Current Snapshot: MvccSnapshot[applied={T|T < 7023594641867259904}]
I20240503 13:59:45.619875 626 sys_catalog.cc:455] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "453b677f710d43a39288ff37d4f050aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaf89f2a0fdd42c3882ef57f112b215e" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 33115 } } peers { permanent_uuid: "d84bcb85a3304a19b7e3f866c0ba5f30" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 46569 } } peers { permanent_uuid: "453b677f710d43a39288ff37d4f050aa" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 40511 } } }
I20240503 13:59:45.619977 626 sys_catalog.cc:458] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:45.621024 878 catalog_manager.cc:1347] Generated new cluster ID: 9fa719aad4394e9f887ac8737d379434
I20240503 13:59:45.621095 878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:45.647470 878 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:45.648645 878 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:45.659070 878 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa: Generated new TSK 0
I20240503 13:59:45.660006 878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:46.429929 622 catalog_manager.cc:1259] Loaded cluster ID: 9fa719aad4394e9f887ac8737d379434
I20240503 13:59:46.430019 622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: loading cluster ID for follower catalog manager: success
I20240503 13:59:46.431404 622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: acquiring CA information for follower catalog manager: success
I20240503 13:59:46.432160 622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 13:59:46.441414 637 catalog_manager.cc:1259] Loaded cluster ID: 9fa719aad4394e9f887ac8737d379434
I20240503 13:59:46.441498 637 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30: loading cluster ID for follower catalog manager: success
I20240503 13:59:46.442667 637 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30: acquiring CA information for follower catalog manager: success
I20240503 13:59:46.443202 637 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 13:59:46.494601 558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43456
I20240503 13:59:46.527228 558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43468
I20240503 13:59:46.557363 558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43480
I20240503 13:59:47.425283 621 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 13:59:47.496068 710 heartbeater.cc:503] Master 127.0.99.60:40511 was elected leader, sending a full tablet report...
I20240503 13:59:47.528172 789 heartbeater.cc:503] Master 127.0.99.60:40511 was elected leader, sending a full tablet report...
I20240503 13:59:47.558158 868 heartbeater.cc:503] Master 127.0.99.60:40511 was elected leader, sending a full tablet report...
I20240503 13:59:47.812316 396 test_util.cc:268] Using random seed: 569720791
I20240503 13:59:47.825521 558 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43490:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 13:59:47.844018 674 tablet_service.cc:1467] Processing CreateTablet for tablet 21e6be00f28548c09f01f2764b04a24d (DEFAULT_TABLE table=test-workload [id=05e035c7c8b24f98905644036f4ef028]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240503 13:59:47.844555 674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e6be00f28548c09f01f2764b04a24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.844859 753 tablet_service.cc:1467] Processing CreateTablet for tablet 8eab4f964c0e4df99954490529c78222 (DEFAULT_TABLE table=test-workload [id=05e035c7c8b24f98905644036f4ef028]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240503 13:59:47.845139 753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eab4f964c0e4df99954490529c78222. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.847100 673 tablet_service.cc:1467] Processing CreateTablet for tablet b4fbe97e2733438d91af4e44c9ac67f0 (DEFAULT_TABLE table=test-workload [id=05e035c7c8b24f98905644036f4ef028]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240503 13:59:47.847329 918 tablet_bootstrap.cc:492] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531: Bootstrap starting.
I20240503 13:59:47.847345 673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4fbe97e2733438d91af4e44c9ac67f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.847765 832 tablet_service.cc:1467] Processing CreateTablet for tablet deebbb9ab5a549e08b2aa768f9de470f (DEFAULT_TABLE table=test-workload [id=05e035c7c8b24f98905644036f4ef028]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240503 13:59:47.848009 832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deebbb9ab5a549e08b2aa768f9de470f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:47.848338 918 tablet_bootstrap.cc:654] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.849364 919 tablet_bootstrap.cc:492] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c: Bootstrap starting.
I20240503 13:59:47.849973 921 tablet_bootstrap.cc:492] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7: Bootstrap starting.
I20240503 13:59:47.850235 919 tablet_bootstrap.cc:654] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.850728 921 tablet_bootstrap.cc:654] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.851310 919 tablet_bootstrap.cc:492] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c: No bootstrap required, opened a new log
I20240503 13:59:47.851310 918 tablet_bootstrap.cc:492] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531: No bootstrap required, opened a new log
I20240503 13:59:47.851440 918 ts_tablet_manager.cc:1367] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:47.851440 919 ts_tablet_manager.cc:1367] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:47.851714 921 tablet_bootstrap.cc:492] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7: No bootstrap required, opened a new log
I20240503 13:59:47.851794 921 ts_tablet_manager.cc:1367] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:47.851935 918 raft_consensus.cc:348] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } }
I20240503 13:59:47.851935 919 raft_consensus.cc:348] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be42e397639b44c99537496653abc38c" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33109 } }
I20240503 13:59:47.852089 918 raft_consensus.cc:374] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.852089 919 raft_consensus.cc:374] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.852108 921 raft_consensus.cc:348] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e7a39f4b1470dad396c6315d191b7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40249 } }
I20240503 13:59:47.852180 918 raft_consensus.cc:729] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d8cff9496741b7b1c324a1a61ac531, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.852201 919 raft_consensus.cc:729] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be42e397639b44c99537496653abc38c, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.852213 921 raft_consensus.cc:374] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.852243 921 raft_consensus.cc:729] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455e7a39f4b1470dad396c6315d191b7, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.852312 918 consensus_queue.cc:260] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } }
I20240503 13:59:47.852322 919 consensus_queue.cc:260] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be42e397639b44c99537496653abc38c" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33109 } }
I20240503 13:59:47.852412 918 raft_consensus.cc:388] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:47.852442 919 raft_consensus.cc:388] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:47.852485 918 raft_consensus.cc:482] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:47.852490 919 raft_consensus.cc:482] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:47.852418 921 consensus_queue.cc:260] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e7a39f4b1470dad396c6315d191b7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40249 } }
I20240503 13:59:47.852586 918 raft_consensus.cc:3037] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.852607 921 raft_consensus.cc:388] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:47.852653 921 raft_consensus.cc:482] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:47.852699 921 raft_consensus.cc:3037] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.852588 919 raft_consensus.cc:3037] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.853440 921 raft_consensus.cc:504] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e7a39f4b1470dad396c6315d191b7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40249 } }
I20240503 13:59:47.853523 918 raft_consensus.cc:504] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } }
I20240503 13:59:47.853562 921 leader_election.cc:302] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 455e7a39f4b1470dad396c6315d191b7; no voters:
I20240503 13:59:47.853559 919 raft_consensus.cc:504] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be42e397639b44c99537496653abc38c" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33109 } }
I20240503 13:59:47.853652 918 leader_election.cc:302] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d8cff9496741b7b1c324a1a61ac531; no voters:
I20240503 13:59:47.853796 921 leader_election.cc:288] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:47.853861 918 leader_election.cc:288] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:47.853884 924 raft_consensus.cc:2781] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:47.854045 924 raft_consensus.cc:686] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 1 LEADER]: Becoming Leader. State: Replica: 455e7a39f4b1470dad396c6315d191b7, State: Running, Role: LEADER
I20240503 13:59:47.854115 921 ts_tablet_manager.cc:1398] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:47.854175 924 consensus_queue.cc:237] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e7a39f4b1470dad396c6315d191b7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40249 } }
I20240503 13:59:47.854449 918 ts_tablet_manager.cc:1398] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:47.857201 557 catalog_manager.cc:5484] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 455e7a39f4b1470dad396c6315d191b7 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "455e7a39f4b1470dad396c6315d191b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "455e7a39f4b1470dad396c6315d191b7" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 40249 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:47.857614 918 tablet_bootstrap.cc:492] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531: Bootstrap starting.
I20240503 13:59:47.858556 918 tablet_bootstrap.cc:654] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:47.859148 919 leader_election.cc:302] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be42e397639b44c99537496653abc38c; no voters:
I20240503 13:59:47.859330 919 leader_election.cc:288] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:47.859592 932 raft_consensus.cc:2781] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:47.859648 919 ts_tablet_manager.cc:1398] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:47.859853 932 raft_consensus.cc:686] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 1 LEADER]: Becoming Leader. State: Replica: be42e397639b44c99537496653abc38c, State: Running, Role: LEADER
I20240503 13:59:47.859979 932 consensus_queue.cc:237] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be42e397639b44c99537496653abc38c" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33109 } }
I20240503 13:59:47.861439 556 catalog_manager.cc:5484] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c reported cstate change: term changed from 0 to 1, leader changed from <none> to be42e397639b44c99537496653abc38c (127.0.99.2). New cstate: current_term: 1 leader_uuid: "be42e397639b44c99537496653abc38c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be42e397639b44c99537496653abc38c" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 33109 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:47.862936 925 raft_consensus.cc:2781] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:47.863373 925 raft_consensus.cc:686] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 1 LEADER]: Becoming Leader. State: Replica: 56d8cff9496741b7b1c324a1a61ac531, State: Running, Role: LEADER
I20240503 13:59:47.863500 925 consensus_queue.cc:237] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } }
I20240503 13:59:47.865159 558 catalog_manager.cc:5484] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56d8cff9496741b7b1c324a1a61ac531 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "56d8cff9496741b7b1c324a1a61ac531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:47.870201 918 tablet_bootstrap.cc:492] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531: No bootstrap required, opened a new log
I20240503 13:59:47.870311 918 ts_tablet_manager.cc:1367] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 13:59:47.870719 918 raft_consensus.cc:348] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } }
I20240503 13:59:47.870810 918 raft_consensus.cc:374] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:47.870841 918 raft_consensus.cc:729] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d8cff9496741b7b1c324a1a61ac531, State: Initialized, Role: FOLLOWER
I20240503 13:59:47.870949 918 consensus_queue.cc:260] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } }
I20240503 13:59:47.871021 918 raft_consensus.cc:388] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:47.871052 918 raft_consensus.cc:482] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:47.871089 918 raft_consensus.cc:3037] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:47.871965 918 raft_consensus.cc:504] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } }
I20240503 13:59:47.872085 918 leader_election.cc:302] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d8cff9496741b7b1c324a1a61ac531; no voters:
I20240503 13:59:47.872184 918 leader_election.cc:288] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:47.872409 918 ts_tablet_manager.cc:1398] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:47.872601 925 raft_consensus.cc:2781] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:47.872689 925 raft_consensus.cc:686] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 1 LEADER]: Becoming Leader. State: Replica: 56d8cff9496741b7b1c324a1a61ac531, State: Running, Role: LEADER
I20240503 13:59:47.872795 925 consensus_queue.cc:237] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } }
I20240503 13:59:47.874222 558 catalog_manager.cc:5484] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56d8cff9496741b7b1c324a1a61ac531 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "56d8cff9496741b7b1c324a1a61ac531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d8cff9496741b7b1c324a1a61ac531" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 45993 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:47.887425 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 13:59:47.891899 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:47.892186 396 tablet_replica.cc:330] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531: stopping tablet replica
I20240503 13:59:47.892452 396 raft_consensus.cc:2229] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:47.892616 396 raft_consensus.cc:2258] T b4fbe97e2733438d91af4e44c9ac67f0 P 56d8cff9496741b7b1c324a1a61ac531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.893807 396 tablet_replica.cc:330] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531: stopping tablet replica
I20240503 13:59:47.893903 396 raft_consensus.cc:2229] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:47.894016 396 raft_consensus.cc:2258] T 21e6be00f28548c09f01f2764b04a24d P 56d8cff9496741b7b1c324a1a61ac531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.908146 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 13:59:47.912756 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 13:59:47.916973 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:47.917265 396 tablet_replica.cc:330] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c: stopping tablet replica
I20240503 13:59:47.917402 396 raft_consensus.cc:2229] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:47.917532 396 raft_consensus.cc:2258] T 8eab4f964c0e4df99954490529c78222 P be42e397639b44c99537496653abc38c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.930693 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 13:59:47.932740 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 13:59:47.936513 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:47.936823 396 tablet_replica.cc:330] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7: stopping tablet replica
I20240503 13:59:47.936957 396 raft_consensus.cc:2229] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:47.937086 396 raft_consensus.cc:2258] T deebbb9ab5a549e08b2aa768f9de470f P 455e7a39f4b1470dad396c6315d191b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.950165 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 13:59:47.952277 396 master.cc:537] Master@127.0.99.62:33115 shutting down...
I20240503 13:59:47.955782 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:47.955978 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.956040 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P aaf89f2a0fdd42c3882ef57f112b215e: stopping tablet replica
I20240503 13:59:47.967466 396 master.cc:552] Master@127.0.99.62:33115 shutdown complete.
I20240503 13:59:47.970361 396 master.cc:537] Master@127.0.99.61:46569 shutting down...
I20240503 13:59:47.973265 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:47.973459 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.973527 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P d84bcb85a3304a19b7e3f866c0ba5f30: stopping tablet replica
I20240503 13:59:47.986654 396 master.cc:552] Master@127.0.99.61:46569 shutdown complete.
I20240503 13:59:47.989423 396 master.cc:537] Master@127.0.99.60:40511 shutting down...
I20240503 13:59:47.993690 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:47.994041 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:47.994117 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 453b677f710d43a39288ff37d4f050aa: stopping tablet replica
I20240503 13:59:48.007201 396 master.cc:552] Master@127.0.99.60:40511 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2738 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240503 13:59:48.015599 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:46765
I20240503 13:59:48.016005 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:48.018460 941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:48.019306 942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:48.019948 944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:48.020467 396 server_base.cc:1034] running on GCE node
I20240503 13:59:48.020664 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:48.020704 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:48.020723 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744788020723 us; error 0 us; skew 500 ppm
I20240503 13:59:48.021757 396 webserver.cc:460] Webserver started at http://127.0.99.62:37659/ using document root <none> and password file <none>
I20240503 13:59:48.021919 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:48.021973 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:48.022061 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:48.022485 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "e77e718e8c6046cc9b6f849f4622706f"
format_stamp: "Formatted at 2024-05-03 13:59:48 on dist-test-slave-6w8v"
I20240503 13:59:48.024048 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:48.025141 949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.025496 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:48.025590 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "e77e718e8c6046cc9b6f849f4622706f"
format_stamp: "Formatted at 2024-05-03 13:59:48 on dist-test-slave-6w8v"
I20240503 13:59:48.025676 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:48.032151 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:48.032575 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:48.037066 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:46765
I20240503 13:59:48.039235 1001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:46765 every 8 connection(s)
I20240503 13:59:48.039891 1002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:48.042330 1002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f: Bootstrap starting.
I20240503 13:59:48.043304 1002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:48.044833 1002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f: No bootstrap required, opened a new log
I20240503 13:59:48.045323 1002 raft_consensus.cc:348] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77e718e8c6046cc9b6f849f4622706f" member_type: VOTER }
I20240503 13:59:48.045439 1002 raft_consensus.cc:374] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:48.045459 1002 raft_consensus.cc:729] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77e718e8c6046cc9b6f849f4622706f, State: Initialized, Role: FOLLOWER
I20240503 13:59:48.045598 1002 consensus_queue.cc:260] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77e718e8c6046cc9b6f849f4622706f" member_type: VOTER }
I20240503 13:59:48.045670 1002 raft_consensus.cc:388] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:48.045691 1002 raft_consensus.cc:482] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:48.045718 1002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:48.046514 1002 raft_consensus.cc:504] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77e718e8c6046cc9b6f849f4622706f" member_type: VOTER }
I20240503 13:59:48.046665 1002 leader_election.cc:302] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e77e718e8c6046cc9b6f849f4622706f; no voters:
I20240503 13:59:48.046885 1002 leader_election.cc:288] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:48.047044 1005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:48.047272 1005 raft_consensus.cc:686] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 1 LEADER]: Becoming Leader. State: Replica: e77e718e8c6046cc9b6f849f4622706f, State: Running, Role: LEADER
I20240503 13:59:48.047448 1002 sys_catalog.cc:564] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:48.047428 1005 consensus_queue.cc:237] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77e718e8c6046cc9b6f849f4622706f" member_type: VOTER }
I20240503 13:59:48.047955 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e77e718e8c6046cc9b6f849f4622706f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77e718e8c6046cc9b6f849f4622706f" member_type: VOTER } }
I20240503 13:59:48.047998 1007 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e77e718e8c6046cc9b6f849f4622706f. Latest consensus state: current_term: 1 leader_uuid: "e77e718e8c6046cc9b6f849f4622706f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77e718e8c6046cc9b6f849f4622706f" member_type: VOTER } }
I20240503 13:59:48.048081 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:48.048095 1007 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:48.048342 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:48.049394 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:48.049734 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 13:59:48.051477 1010 catalog_manager.cc:1347] Generated new cluster ID: f6bb0025a94f437f83780e27e2796948
I20240503 13:59:48.051551 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:48.064114 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:48.064818 1010 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:48.079074 1010 catalog_manager.cc:5857] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f: Generated new TSK 0
I20240503 13:59:48.079329 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:48.082521 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:48.085642 1024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:48.086428 1023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:48.087234 1026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:48.087872 396 server_base.cc:1034] running on GCE node
I20240503 13:59:48.088088 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:48.088132 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:48.088155 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744788088154 us; error 0 us; skew 500 ppm
I20240503 13:59:48.089393 396 webserver.cc:460] Webserver started at http://127.0.99.1:42671/ using document root <none> and password file <none>
I20240503 13:59:48.089576 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:48.089659 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:48.089749 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:48.090224 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-0-root/instance:
uuid: "176786c72902464ca12fbfb8c6a44767"
format_stamp: "Formatted at 2024-05-03 13:59:48 on dist-test-slave-6w8v"
I20240503 13:59:48.092137 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:48.093377 1031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.093720 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:48.093808 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-0-root
uuid: "176786c72902464ca12fbfb8c6a44767"
format_stamp: "Formatted at 2024-05-03 13:59:48 on dist-test-slave-6w8v"
I20240503 13:59:48.093894 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:48.117478 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:48.118324 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:48.119179 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:48.120075 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:48.120144 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.120194 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:48.120218 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.126184 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:33661
I20240503 13:59:48.127249 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:48.126216 1094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:33661 every 8 connection(s)
W20240503 13:59:48.130419 1100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:48.130887 1102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:48.131995 1099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:48.132447 396 server_base.cc:1034] running on GCE node
I20240503 13:59:48.132678 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:48.132728 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:48.132751 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744788132751 us; error 0 us; skew 500 ppm
I20240503 13:59:48.133198 1095 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46765
I20240503 13:59:48.133306 1095 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:48.133598 1095 heartbeater.cc:511] Master 127.0.99.62:46765 requested a full tablet report, sending...
I20240503 13:59:48.134052 396 webserver.cc:460] Webserver started at http://127.0.99.2:37313/ using document root <none> and password file <none>
I20240503 13:59:48.134224 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:48.134287 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:48.134347 966 ts_manager.cc:194] Registered new tserver with Master: 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661)
I20240503 13:59:48.134369 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:48.134883 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-1-root/instance:
uuid: "68ebad4bc2494af6a689f6d1bf05cc48"
format_stamp: "Formatted at 2024-05-03 13:59:48 on dist-test-slave-6w8v"
I20240503 13:59:48.135422 966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50816
I20240503 13:59:48.136889 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:48.138075 1107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.138394 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:48.138475 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-1-root
uuid: "68ebad4bc2494af6a689f6d1bf05cc48"
format_stamp: "Formatted at 2024-05-03 13:59:48 on dist-test-slave-6w8v"
I20240503 13:59:48.138576 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:48.150916 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:48.151427 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:48.151906 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:48.152516 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:48.152575 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.152628 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:48.152647 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.157233 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:46427
I20240503 13:59:48.157316 1170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:46427 every 8 connection(s)
I20240503 13:59:48.158152 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:48.160924 1175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:48.161849 1174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:48.162519 1177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:48.162833 1171 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46765
I20240503 13:59:48.162956 1171 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:48.162982 396 server_base.cc:1034] running on GCE node
I20240503 13:59:48.163174 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:48.163213 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:48.163232 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744788163232 us; error 0 us; skew 500 ppm
I20240503 13:59:48.163340 1171 heartbeater.cc:511] Master 127.0.99.62:46765 requested a full tablet report, sending...
I20240503 13:59:48.164049 966 ts_manager.cc:194] Registered new tserver with Master: 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427)
I20240503 13:59:48.164451 396 webserver.cc:460] Webserver started at http://127.0.99.3:35607/ using document root <none> and password file <none>
I20240503 13:59:48.164598 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:48.164650 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:48.164688 966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50832
I20240503 13:59:48.164728 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:48.165160 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-2-root/instance:
uuid: "afbf7d3c38d34497b671d263eef45878"
format_stamp: "Formatted at 2024-05-03 13:59:48 on dist-test-slave-6w8v"
I20240503 13:59:48.166671 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:48.167770 1182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.168042 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:48.168110 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-2-root
uuid: "afbf7d3c38d34497b671d263eef45878"
format_stamp: "Formatted at 2024-05-03 13:59:48 on dist-test-slave-6w8v"
I20240503 13:59:48.168169 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:48.185793 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:48.186372 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:48.186694 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:48.187186 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:48.187230 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.187263 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:48.187281 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:48.191849 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:46843
I20240503 13:59:48.191922 1245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:46843 every 8 connection(s)
I20240503 13:59:48.196656 1246 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46765
I20240503 13:59:48.196846 1246 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:48.197140 1246 heartbeater.cc:511] Master 127.0.99.62:46765 requested a full tablet report, sending...
I20240503 13:59:48.197813 966 ts_manager.cc:194] Registered new tserver with Master: afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:48.197907 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005663459s
I20240503 13:59:48.198405 966 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50846
I20240503 13:59:49.136759 1095 heartbeater.cc:503] Master 127.0.99.62:46765 was elected leader, sending a full tablet report...
I20240503 13:59:49.165656 1171 heartbeater.cc:503] Master 127.0.99.62:46765 was elected leader, sending a full tablet report...
I20240503 13:59:49.199469 1246 heartbeater.cc:503] Master 127.0.99.62:46765 was elected leader, sending a full tablet report...
I20240503 13:59:49.224752 396 test_util.cc:268] Using random seed: 571133228
I20240503 13:59:49.233937 966 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50848:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 13:59:49.234583 966 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 13:59:49.248692 1059 tablet_service.cc:1467] Processing CreateTablet for tablet c9532ea79b8e4d518c3ea78908ed0765 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 13:59:49.249157 1059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9532ea79b8e4d518c3ea78908ed0765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.249060 1058 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd7c505bbd54343bdcccc2b0f7d05ae (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 13:59:49.249405 1054 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0e51b991cd4846920981e8274ff08b (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 13:59:49.249488 1052 tablet_service.cc:1467] Processing CreateTablet for tablet f4c1cd1783c14c1e9cd7deb2512e6584 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 13:59:49.249577 1058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd7c505bbd54343bdcccc2b0f7d05ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.249718 1052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c1cd1783c14c1e9cd7deb2512e6584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.249821 1055 tablet_service.cc:1467] Processing CreateTablet for tablet df636cac17de48aba6a1f0b2927d6e6d (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 13:59:49.250054 1055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df636cac17de48aba6a1f0b2927d6e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.251377 1056 tablet_service.cc:1467] Processing CreateTablet for tablet 06c5d53b0cf445c19ca0a49da5e70fb0 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 13:59:49.251667 1210 tablet_service.cc:1467] Processing CreateTablet for tablet c9532ea79b8e4d518c3ea78908ed0765 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 13:59:49.249060 1053 tablet_service.cc:1467] Processing CreateTablet for tablet e4484a16802b4141a38cd0008bbddecc (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 13:59:49.251940 1210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9532ea79b8e4d518c3ea78908ed0765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.253887 1203 tablet_service.cc:1467] Processing CreateTablet for tablet f4c1cd1783c14c1e9cd7deb2512e6584 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 13:59:49.254004 1133 tablet_service.cc:1467] Processing CreateTablet for tablet 8380321f903c4ffaa86ba33d02c09906 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 13:59:49.254176 1203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c1cd1783c14c1e9cd7deb2512e6584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.254248 1133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8380321f903c4ffaa86ba33d02c09906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.254225 1204 tablet_service.cc:1467] Processing CreateTablet for tablet e4484a16802b4141a38cd0008bbddecc (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 13:59:49.254348 1267 tablet_bootstrap.cc:492] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878: Bootstrap starting.
I20240503 13:59:49.254446 1204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4484a16802b4141a38cd0008bbddecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.255585 1267 tablet_bootstrap.cc:654] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.256469 1205 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0e51b991cd4846920981e8274ff08b (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 13:59:49.256709 1205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0e51b991cd4846920981e8274ff08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.256731 1127 tablet_service.cc:1467] Processing CreateTablet for tablet f4c1cd1783c14c1e9cd7deb2512e6584 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240503 13:59:49.256814 1267 tablet_bootstrap.cc:492] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878: No bootstrap required, opened a new log
I20240503 13:59:49.256901 1267 ts_tablet_manager.cc:1367] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:49.256947 1127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4c1cd1783c14c1e9cd7deb2512e6584. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.257125 1269 tablet_bootstrap.cc:492] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48: Bootstrap starting.
I20240503 13:59:49.257385 1267 raft_consensus.cc:348] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.257494 1267 raft_consensus.cc:374] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.257529 1267 raft_consensus.cc:729] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.257658 1267 consensus_queue.cc:260] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.258013 1267 ts_tablet_manager.cc:1398] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.258208 1267 tablet_bootstrap.cc:492] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878: Bootstrap starting.
I20240503 13:59:49.258734 1269 tablet_bootstrap.cc:654] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.258895 1206 tablet_service.cc:1467] Processing CreateTablet for tablet df636cac17de48aba6a1f0b2927d6e6d (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 13:59:49.259052 1267 tablet_bootstrap.cc:654] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.259138 1206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df636cac17de48aba6a1f0b2927d6e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.256441 1129 tablet_service.cc:1467] Processing CreateTablet for tablet 06c5d53b0cf445c19ca0a49da5e70fb0 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 13:59:49.259709 1129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c5d53b0cf445c19ca0a49da5e70fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.260468 1267 tablet_bootstrap.cc:492] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878: No bootstrap required, opened a new log
I20240503 13:59:49.260468 1269 tablet_bootstrap.cc:492] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48: No bootstrap required, opened a new log
I20240503 13:59:49.260566 1267 ts_tablet_manager.cc:1367] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 13:59:49.260584 1269 ts_tablet_manager.cc:1367] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 13:59:49.261008 1267 raft_consensus.cc:348] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.261119 1267 raft_consensus.cc:374] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.261152 1267 raft_consensus.cc:729] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.261142 1269 raft_consensus.cc:348] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.261278 1269 raft_consensus.cc:374] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.261319 1267 consensus_queue.cc:260] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.261356 1269 raft_consensus.cc:729] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.261582 1267 ts_tablet_manager.cc:1398] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.261525 1208 tablet_service.cc:1467] Processing CreateTablet for tablet 8380321f903c4ffaa86ba33d02c09906 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 13:59:49.261538 1269 consensus_queue.cc:260] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.261752 1267 tablet_bootstrap.cc:492] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878: Bootstrap starting.
I20240503 13:59:49.261309 1207 tablet_service.cc:1467] Processing CreateTablet for tablet 06c5d53b0cf445c19ca0a49da5e70fb0 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240503 13:59:49.261812 1208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8380321f903c4ffaa86ba33d02c09906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.262020 1209 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd7c505bbd54343bdcccc2b0f7d05ae (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 13:59:49.262166 1269 ts_tablet_manager.cc:1398] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 13:59:49.262287 1209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd7c505bbd54343bdcccc2b0f7d05ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.262439 1207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c5d53b0cf445c19ca0a49da5e70fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.262701 1135 tablet_service.cc:1467] Processing CreateTablet for tablet c9532ea79b8e4d518c3ea78908ed0765 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240503 13:59:49.262933 1135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9532ea79b8e4d518c3ea78908ed0765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.262954 1269 tablet_bootstrap.cc:492] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48: Bootstrap starting.
I20240503 13:59:49.264562 1269 tablet_bootstrap.cc:654] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.265049 1134 tablet_service.cc:1467] Processing CreateTablet for tablet 3fd7c505bbd54343bdcccc2b0f7d05ae (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240503 13:59:49.265341 1134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fd7c505bbd54343bdcccc2b0f7d05ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.266422 1269 tablet_bootstrap.cc:492] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48: No bootstrap required, opened a new log
I20240503 13:59:49.266510 1269 ts_tablet_manager.cc:1367] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 13:59:49.266934 1269 raft_consensus.cc:348] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.267048 1269 raft_consensus.cc:374] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.267081 1269 raft_consensus.cc:729] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.267138 1267 tablet_bootstrap.cc:654] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.267203 1269 consensus_queue.cc:260] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.267458 1132 tablet_service.cc:1467] Processing CreateTablet for tablet df636cac17de48aba6a1f0b2927d6e6d (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240503 13:59:49.267614 1269 ts_tablet_manager.cc:1398] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:49.267741 1132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df636cac17de48aba6a1f0b2927d6e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.267815 1269 tablet_bootstrap.cc:492] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48: Bootstrap starting.
I20240503 13:59:49.268291 1267 tablet_bootstrap.cc:492] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878: No bootstrap required, opened a new log
I20240503 13:59:49.268383 1267 ts_tablet_manager.cc:1367] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240503 13:59:49.268821 1267 raft_consensus.cc:348] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.268914 1267 raft_consensus.cc:374] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.268946 1267 raft_consensus.cc:729] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.269057 1267 consensus_queue.cc:260] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.269338 1267 ts_tablet_manager.cc:1398] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:49.269510 1267 tablet_bootstrap.cc:492] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878: Bootstrap starting.
I20240503 13:59:49.269644 1269 tablet_bootstrap.cc:654] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.269785 1131 tablet_service.cc:1467] Processing CreateTablet for tablet 6a0e51b991cd4846920981e8274ff08b (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240503 13:59:49.270090 1131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0e51b991cd4846920981e8274ff08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.270449 1267 tablet_bootstrap.cc:654] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.271473 1267 tablet_bootstrap.cc:492] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878: No bootstrap required, opened a new log
I20240503 13:59:49.271473 1269 tablet_bootstrap.cc:492] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48: No bootstrap required, opened a new log
I20240503 13:59:49.271603 1267 ts_tablet_manager.cc:1367] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:49.271617 1269 ts_tablet_manager.cc:1367] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 13:59:49.272040 1267 raft_consensus.cc:348] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.272153 1267 raft_consensus.cc:374] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.272186 1267 raft_consensus.cc:729] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.272226 1130 tablet_service.cc:1467] Processing CreateTablet for tablet e4484a16802b4141a38cd0008bbddecc (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240503 13:59:49.272405 1267 consensus_queue.cc:260] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.248692 1057 tablet_service.cc:1467] Processing CreateTablet for tablet 8380321f903c4ffaa86ba33d02c09906 (DEFAULT_TABLE table=test-workload [id=365b84c76658445892e782611df258b6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240503 13:59:49.272718 1267 ts_tablet_manager.cc:1398] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.251940 1054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a0e51b991cd4846920981e8274ff08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.272873 1267 tablet_bootstrap.cc:492] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878: Bootstrap starting.
I20240503 13:59:49.273759 1267 tablet_bootstrap.cc:654] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.274448 1266 tablet_bootstrap.cc:492] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767: Bootstrap starting.
I20240503 13:59:49.274842 1130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4484a16802b4141a38cd0008bbddecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.274940 1056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c5d53b0cf445c19ca0a49da5e70fb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.275398 1266 tablet_bootstrap.cc:654] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.275651 1053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4484a16802b4141a38cd0008bbddecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.276811 1266 tablet_bootstrap.cc:492] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767: No bootstrap required, opened a new log
I20240503 13:59:49.276898 1266 ts_tablet_manager.cc:1367] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:49.277384 1266 raft_consensus.cc:348] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.277496 1266 raft_consensus.cc:374] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.277529 1266 raft_consensus.cc:729] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.277536 1057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8380321f903c4ffaa86ba33d02c09906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:49.277670 1266 consensus_queue.cc:260] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.278034 1266 ts_tablet_manager.cc:1398] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:49.278213 1266 tablet_bootstrap.cc:492] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767: Bootstrap starting.
I20240503 13:59:49.272222 1269 raft_consensus.cc:348] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.278384 1269 raft_consensus.cc:374] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.278424 1269 raft_consensus.cc:729] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.278539 1269 consensus_queue.cc:260] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.278785 1269 ts_tablet_manager.cc:1398] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 13:59:49.278944 1269 tablet_bootstrap.cc:492] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48: Bootstrap starting.
I20240503 13:59:49.279151 1266 tablet_bootstrap.cc:654] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.280042 1269 tablet_bootstrap.cc:654] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.281137 1267 tablet_bootstrap.cc:492] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878: No bootstrap required, opened a new log
I20240503 13:59:49.281235 1267 ts_tablet_manager.cc:1367] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 13:59:49.281666 1267 raft_consensus.cc:348] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.281795 1267 raft_consensus.cc:374] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.281831 1267 raft_consensus.cc:729] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.281916 1269 tablet_bootstrap.cc:492] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48: No bootstrap required, opened a new log
I20240503 13:59:49.281996 1269 ts_tablet_manager.cc:1367] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:49.281949 1267 consensus_queue.cc:260] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.282240 1267 ts_tablet_manager.cc:1398] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.282407 1267 tablet_bootstrap.cc:492] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878: Bootstrap starting.
I20240503 13:59:49.282428 1269 raft_consensus.cc:348] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.282526 1269 raft_consensus.cc:374] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.282553 1269 raft_consensus.cc:729] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.282655 1269 consensus_queue.cc:260] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.282912 1269 ts_tablet_manager.cc:1398] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:49.282959 1266 tablet_bootstrap.cc:492] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767: No bootstrap required, opened a new log
I20240503 13:59:49.283032 1266 ts_tablet_manager.cc:1367] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:49.283051 1269 tablet_bootstrap.cc:492] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48: Bootstrap starting.
I20240503 13:59:49.283356 1267 tablet_bootstrap.cc:654] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.283413 1266 raft_consensus.cc:348] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.283504 1266 raft_consensus.cc:374] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.283546 1266 raft_consensus.cc:729] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.283699 1266 consensus_queue.cc:260] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.283950 1266 ts_tablet_manager.cc:1398] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.284004 1269 tablet_bootstrap.cc:654] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.284109 1266 tablet_bootstrap.cc:492] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767: Bootstrap starting.
I20240503 13:59:49.285013 1266 tablet_bootstrap.cc:654] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.285269 1269 tablet_bootstrap.cc:492] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48: No bootstrap required, opened a new log
I20240503 13:59:49.285357 1269 ts_tablet_manager.cc:1367] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:49.285790 1267 tablet_bootstrap.cc:492] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878: No bootstrap required, opened a new log
I20240503 13:59:49.285801 1269 raft_consensus.cc:348] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.285939 1269 raft_consensus.cc:374] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.285979 1269 raft_consensus.cc:729] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.286073 1267 ts_tablet_manager.cc:1367] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:49.286103 1269 consensus_queue.cc:260] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.286383 1269 ts_tablet_manager.cc:1398] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.286587 1269 tablet_bootstrap.cc:492] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48: Bootstrap starting.
I20240503 13:59:49.286712 1267 raft_consensus.cc:348] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.286893 1267 raft_consensus.cc:374] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.286971 1267 raft_consensus.cc:729] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.287155 1267 consensus_queue.cc:260] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.287514 1267 ts_tablet_manager.cc:1398] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.287775 1267 tablet_bootstrap.cc:492] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878: Bootstrap starting.
I20240503 13:59:49.287542 1269 tablet_bootstrap.cc:654] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.287871 1266 tablet_bootstrap.cc:492] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767: No bootstrap required, opened a new log
I20240503 13:59:49.287946 1266 ts_tablet_manager.cc:1367] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240503 13:59:49.288404 1266 raft_consensus.cc:348] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.288498 1266 raft_consensus.cc:374] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.288522 1266 raft_consensus.cc:729] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.288643 1266 consensus_queue.cc:260] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.288861 1266 ts_tablet_manager.cc:1398] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:49.289016 1267 tablet_bootstrap.cc:654] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.289037 1266 tablet_bootstrap.cc:492] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767: Bootstrap starting.
I20240503 13:59:49.290333 1266 tablet_bootstrap.cc:654] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.290490 1267 tablet_bootstrap.cc:492] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878: No bootstrap required, opened a new log
I20240503 13:59:49.290570 1267 ts_tablet_manager.cc:1367] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:49.291277 1267 raft_consensus.cc:348] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.291388 1267 raft_consensus.cc:374] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.291419 1267 raft_consensus.cc:729] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.291528 1266 tablet_bootstrap.cc:492] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767: No bootstrap required, opened a new log
I20240503 13:59:49.291671 1266 ts_tablet_manager.cc:1367] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:49.291538 1267 consensus_queue.cc:260] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.291720 1269 tablet_bootstrap.cc:492] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48: No bootstrap required, opened a new log
I20240503 13:59:49.291803 1269 ts_tablet_manager.cc:1367] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 13:59:49.292012 1267 ts_tablet_manager.cc:1398] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.292172 1267 tablet_bootstrap.cc:492] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878: Bootstrap starting.
I20240503 13:59:49.292214 1269 raft_consensus.cc:348] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.292315 1269 raft_consensus.cc:374] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.292349 1269 raft_consensus.cc:729] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.292397 1270 raft_consensus.cc:482] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.292452 1269 consensus_queue.cc:260] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.292480 1270 raft_consensus.cc:504] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.292701 1269 ts_tablet_manager.cc:1398] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.292865 1269 tablet_bootstrap.cc:492] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48: Bootstrap starting.
I20240503 13:59:49.292968 1270 leader_election.cc:288] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427)
I20240503 13:59:49.293152 1267 tablet_bootstrap.cc:654] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.293761 1269 tablet_bootstrap.cc:654] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.294414 1267 tablet_bootstrap.cc:492] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878: No bootstrap required, opened a new log
I20240503 13:59:49.294514 1267 ts_tablet_manager.cc:1367] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:49.294965 1267 raft_consensus.cc:348] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.295056 1269 tablet_bootstrap.cc:492] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48: No bootstrap required, opened a new log
I20240503 13:59:49.295066 1267 raft_consensus.cc:374] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.295121 1267 raft_consensus.cc:729] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.295142 1269 ts_tablet_manager.cc:1367] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:49.295257 1267 consensus_queue.cc:260] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.295526 1267 ts_tablet_manager.cc:1398] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.295544 1269 raft_consensus.cc:348] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.295676 1269 raft_consensus.cc:374] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.295713 1269 raft_consensus.cc:729] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.295850 1269 consensus_queue.cc:260] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.296171 1269 ts_tablet_manager.cc:1398] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:49.296358 1269 tablet_bootstrap.cc:492] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48: Bootstrap starting.
I20240503 13:59:49.297278 1269 tablet_bootstrap.cc:654] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.292119 1266 raft_consensus.cc:348] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.297767 1266 raft_consensus.cc:374] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.297830 1266 raft_consensus.cc:729] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.297951 1266 consensus_queue.cc:260] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.298247 1266 ts_tablet_manager.cc:1398] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240503 13:59:49.298435 1266 tablet_bootstrap.cc:492] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767: Bootstrap starting.
I20240503 13:59:49.299120 1269 tablet_bootstrap.cc:492] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48: No bootstrap required, opened a new log
I20240503 13:59:49.299208 1269 ts_tablet_manager.cc:1367] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 13:59:49.299223 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0e51b991cd4846920981e8274ff08b" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" is_pre_election: true
I20240503 13:59:49.299347 1266 tablet_bootstrap.cc:654] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.299389 1145 raft_consensus.cc:2454] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 0.
I20240503 13:59:49.299680 1269 raft_consensus.cc:348] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.299793 1269 raft_consensus.cc:374] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.299775 1186 leader_election.cc:302] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ebad4bc2494af6a689f6d1bf05cc48, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.299847 1269 raft_consensus.cc:729] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.299980 1269 consensus_queue.cc:260] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.299988 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0e51b991cd4846920981e8274ff08b" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767" is_pre_election: true
I20240503 13:59:49.300077 1270 raft_consensus.cc:2781] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.300122 1270 raft_consensus.cc:482] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.300155 1270 raft_consensus.cc:3037] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.300283 1269 ts_tablet_manager.cc:1398] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240503 13:59:49.300539 1184 leader_election.cc:341] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661): Illegal state: must be running to vote when last-logged opid is not known
I20240503 13:59:49.302284 1266 tablet_bootstrap.cc:492] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767: No bootstrap required, opened a new log
I20240503 13:59:49.302408 1266 ts_tablet_manager.cc:1367] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 13:59:49.302894 1266 raft_consensus.cc:348] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.303018 1266 raft_consensus.cc:374] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.303048 1266 raft_consensus.cc:729] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.303148 1266 consensus_queue.cc:260] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.303369 1266 ts_tablet_manager.cc:1398] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:49.303620 1266 tablet_bootstrap.cc:492] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767: Bootstrap starting.
I20240503 13:59:49.304682 1266 tablet_bootstrap.cc:654] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.305817 1266 tablet_bootstrap.cc:492] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767: No bootstrap required, opened a new log
I20240503 13:59:49.305915 1266 ts_tablet_manager.cc:1367] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:49.306317 1266 raft_consensus.cc:348] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.306418 1266 raft_consensus.cc:374] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.306447 1266 raft_consensus.cc:729] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.306566 1266 consensus_queue.cc:260] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.306807 1266 ts_tablet_manager.cc:1398] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 13:59:49.306977 1266 tablet_bootstrap.cc:492] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767: Bootstrap starting.
I20240503 13:59:49.307951 1266 tablet_bootstrap.cc:654] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.308012 1270 raft_consensus.cc:504] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.308419 1270 leader_election.cc:288] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 election: Requested vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427)
I20240503 13:59:49.308727 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0e51b991cd4846920981e8274ff08b" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767"
I20240503 13:59:49.308790 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a0e51b991cd4846920981e8274ff08b" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48"
I20240503 13:59:49.308838 1069 raft_consensus.cc:3037] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.308890 1145 raft_consensus.cc:3037] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.309415 1266 tablet_bootstrap.cc:492] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767: No bootstrap required, opened a new log
I20240503 13:59:49.309499 1266 ts_tablet_manager.cc:1367] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:49.309888 1145 raft_consensus.cc:2454] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 1.
I20240503 13:59:49.309957 1266 raft_consensus.cc:348] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.310043 1266 raft_consensus.cc:374] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.310074 1266 raft_consensus.cc:729] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.310189 1266 consensus_queue.cc:260] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.310218 1186 leader_election.cc:302] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ebad4bc2494af6a689f6d1bf05cc48, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.309888 1069 raft_consensus.cc:2454] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 1.
I20240503 13:59:49.310447 1266 ts_tablet_manager.cc:1398] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.310525 1270 raft_consensus.cc:2781] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.310647 1266 tablet_bootstrap.cc:492] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767: Bootstrap starting.
I20240503 13:59:49.310714 1270 raft_consensus.cc:686] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 1 LEADER]: Becoming Leader. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Running, Role: LEADER
I20240503 13:59:49.310832 1270 consensus_queue.cc:237] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.311610 1266 tablet_bootstrap.cc:654] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:49.312896 1266 tablet_bootstrap.cc:492] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767: No bootstrap required, opened a new log
I20240503 13:59:49.312994 1266 ts_tablet_manager.cc:1367] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 13:59:49.313367 1266 raft_consensus.cc:348] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.313441 964 catalog_manager.cc:5484] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 reported cstate change: term changed from 0 to 1, leader changed from <none> to afbf7d3c38d34497b671d263eef45878 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "afbf7d3c38d34497b671d263eef45878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:49.313458 1266 raft_consensus.cc:374] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:49.313742 1266 raft_consensus.cc:729] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Initialized, Role: FOLLOWER
I20240503 13:59:49.313874 1266 consensus_queue.cc:260] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.314146 1266 ts_tablet_manager.cc:1398] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:49.408856 1281 raft_consensus.cc:482] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.409018 1281 raft_consensus.cc:504] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.409579 1281 leader_election.cc:288] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.414136 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df636cac17de48aba6a1f0b2927d6e6d" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" is_pre_election: true
I20240503 13:59:49.414141 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df636cac17de48aba6a1f0b2927d6e6d" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878" is_pre_election: true
I20240503 13:59:49.414369 1145 raft_consensus.cc:2454] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 0.
I20240503 13:59:49.414383 1220 raft_consensus.cc:2454] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 0.
I20240503 13:59:49.414716 1035 leader_election.cc:302] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, 68ebad4bc2494af6a689f6d1bf05cc48, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.414901 1281 raft_consensus.cc:2781] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.414958 1281 raft_consensus.cc:482] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.414984 1281 raft_consensus.cc:3037] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.422861 1303 raft_consensus.cc:482] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.423017 1303 raft_consensus.cc:504] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.423482 1303 leader_election.cc:288] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.423944 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9532ea79b8e4d518c3ea78908ed0765" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" is_pre_election: true
I20240503 13:59:49.423985 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9532ea79b8e4d518c3ea78908ed0765" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878" is_pre_election: true
I20240503 13:59:49.424096 1220 raft_consensus.cc:2454] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 0.
I20240503 13:59:49.424098 1145 raft_consensus.cc:2454] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 0.
I20240503 13:59:49.424755 1032 leader_election.cc:302] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.424963 1303 raft_consensus.cc:2781] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.425040 1303 raft_consensus.cc:482] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.425079 1303 raft_consensus.cc:3037] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.445930 1275 raft_consensus.cc:482] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.446094 1275 raft_consensus.cc:504] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.446640 1275 leader_election.cc:288] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.450182 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c5d53b0cf445c19ca0a49da5e70fb0" candidate_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878" is_pre_election: true
I20240503 13:59:49.450183 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c5d53b0cf445c19ca0a49da5e70fb0" candidate_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767" is_pre_election: true
I20240503 13:59:49.450368 1220 raft_consensus.cc:2454] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ebad4bc2494af6a689f6d1bf05cc48 in term 0.
I20240503 13:59:49.450368 1069 raft_consensus.cc:2454] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ebad4bc2494af6a689f6d1bf05cc48 in term 0.
I20240503 13:59:49.450659 1109 leader_election.cc:302] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, 68ebad4bc2494af6a689f6d1bf05cc48; no voters:
I20240503 13:59:49.450834 1275 raft_consensus.cc:2781] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.450896 1275 raft_consensus.cc:482] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.450939 1275 raft_consensus.cc:3037] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.459012 1298 raft_consensus.cc:482] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.459187 1298 raft_consensus.cc:504] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.459719 1298 leader_election.cc:288] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427)
I20240503 13:59:49.460111 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8380321f903c4ffaa86ba33d02c09906" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767" is_pre_election: true
I20240503 13:59:49.460119 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8380321f903c4ffaa86ba33d02c09906" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" is_pre_election: true
I20240503 13:59:49.460268 1069 raft_consensus.cc:2454] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 0.
I20240503 13:59:49.460278 1145 raft_consensus.cc:2454] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 0.
I20240503 13:59:49.460575 1184 leader_election.cc:302] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.460803 1298 raft_consensus.cc:2781] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.460862 1298 raft_consensus.cc:482] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.460901 1298 raft_consensus.cc:3037] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.462090 1306 raft_consensus.cc:482] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.462181 1306 raft_consensus.cc:504] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.462561 1306 leader_election.cc:288] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.462895 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4484a16802b4141a38cd0008bbddecc" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878" is_pre_election: true
I20240503 13:59:49.462895 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4484a16802b4141a38cd0008bbddecc" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" is_pre_election: true
I20240503 13:59:49.463048 1220 raft_consensus.cc:2454] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 0.
I20240503 13:59:49.463048 1145 raft_consensus.cc:2454] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 0.
I20240503 13:59:49.463323 1032 leader_election.cc:302] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, 68ebad4bc2494af6a689f6d1bf05cc48, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.463482 1306 raft_consensus.cc:2781] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.463543 1306 raft_consensus.cc:482] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.463585 1306 raft_consensus.cc:3037] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.509630 1275 raft_consensus.cc:504] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.510258 1275 leader_election.cc:288] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [CANDIDATE]: Term 1 election: Requested vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.509630 1281 raft_consensus.cc:504] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.510617 1281 leader_election.cc:288] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 election: Requested vote from peers 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.510730 1306 raft_consensus.cc:504] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.510814 1303 raft_consensus.cc:504] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.511025 1306 leader_election.cc:288] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 election: Requested vote from peers 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.511157 1303 leader_election.cc:288] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 election: Requested vote from peers 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.511730 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c5d53b0cf445c19ca0a49da5e70fb0" candidate_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767"
I20240503 13:59:49.511782 1145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df636cac17de48aba6a1f0b2927d6e6d" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48"
I20240503 13:59:49.511729 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4484a16802b4141a38cd0008bbddecc" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48"
I20240503 13:59:49.511909 1069 raft_consensus.cc:3037] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.511943 1144 raft_consensus.cc:3037] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.512019 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9532ea79b8e4d518c3ea78908ed0765" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878"
I20240503 13:59:49.512094 1218 raft_consensus.cc:3037] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.511888 1217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c5d53b0cf445c19ca0a49da5e70fb0" candidate_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878"
I20240503 13:59:49.512836 1144 raft_consensus.cc:2454] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 1.
I20240503 13:59:49.512895 1217 raft_consensus.cc:3037] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.512988 1218 raft_consensus.cc:2454] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 1.
I20240503 13:59:49.511916 1145 raft_consensus.cc:3037] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.513199 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4484a16802b4141a38cd0008bbddecc" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878"
I20240503 13:59:49.513226 1143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9532ea79b8e4d518c3ea78908ed0765" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48"
I20240503 13:59:49.513309 1143 raft_consensus.cc:3037] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.513578 1032 leader_election.cc:302] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.509630 1298 raft_consensus.cc:504] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.513839 1220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df636cac17de48aba6a1f0b2927d6e6d" candidate_uuid: "176786c72902464ca12fbfb8c6a44767" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878"
I20240503 13:59:49.513931 1220 raft_consensus.cc:3037] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.513837 1035 leader_election.cc:302] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, 68ebad4bc2494af6a689f6d1bf05cc48; no voters:
I20240503 13:59:49.513309 1219 raft_consensus.cc:3037] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.514204 1298 leader_election.cc:288] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 election: Requested vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427)
I20240503 13:59:49.514230 1306 raft_consensus.cc:2781] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.514392 1306 raft_consensus.cc:686] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 1 LEADER]: Becoming Leader. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Running, Role: LEADER
I20240503 13:59:49.512836 1069 raft_consensus.cc:2454] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ebad4bc2494af6a689f6d1bf05cc48 in term 1.
I20240503 13:59:49.513757 1303 raft_consensus.cc:2781] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.514540 1303 raft_consensus.cc:686] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 1 LEADER]: Becoming Leader. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Running, Role: LEADER
I20240503 13:59:49.514689 1303 consensus_queue.cc:237] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } }
I20240503 13:59:49.514714 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8380321f903c4ffaa86ba33d02c09906" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767"
I20240503 13:59:49.514714 1306 consensus_queue.cc:237] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.514798 1109 leader_election.cc:302] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, 68ebad4bc2494af6a689f6d1bf05cc48; no voters:
I20240503 13:59:49.514952 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8380321f903c4ffaa86ba33d02c09906" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48"
I20240503 13:59:49.515033 1144 raft_consensus.cc:3037] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.515093 1273 raft_consensus.cc:482] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.515149 1273 raft_consensus.cc:504] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.515200 1275 raft_consensus.cc:2781] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.515362 1275 raft_consensus.cc:686] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 LEADER]: Becoming Leader. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Running, Role: LEADER
I20240503 13:59:49.515470 1273 leader_election.cc:288] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.515478 1275 consensus_queue.cc:237] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.516142 1218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd7c505bbd54343bdcccc2b0f7d05ae" candidate_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878" is_pre_election: true
I20240503 13:59:49.516238 1218 raft_consensus.cc:2454] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ebad4bc2494af6a689f6d1bf05cc48 in term 0.
I20240503 13:59:49.514813 1069 raft_consensus.cc:3037] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.516520 966 catalog_manager.cc:5484] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176786c72902464ca12fbfb8c6a44767 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "176786c72902464ca12fbfb8c6a44767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:49.517060 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd7c505bbd54343bdcccc2b0f7d05ae" candidate_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767" is_pre_election: true
I20240503 13:59:49.517163 1068 raft_consensus.cc:2454] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ebad4bc2494af6a689f6d1bf05cc48 in term 0.
I20240503 13:59:49.517385 1298 raft_consensus.cc:482] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:49.517431 1108 leader_election.cc:302] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ebad4bc2494af6a689f6d1bf05cc48, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.517460 1298 raft_consensus.cc:504] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.517397 964 catalog_manager.cc:5484] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:49.517722 1275 raft_consensus.cc:2781] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.517772 1275 raft_consensus.cc:482] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.517808 1275 raft_consensus.cc:3037] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.517822 1298 leader_election.cc:288] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427)
I20240503 13:59:49.518173 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c1cd1783c14c1e9cd7deb2512e6584" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767" is_pre_election: true
I20240503 13:59:49.518285 1142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c1cd1783c14c1e9cd7deb2512e6584" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" is_pre_election: true
I20240503 13:59:49.518397 1142 raft_consensus.cc:2454] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 0.
I20240503 13:59:49.518286 1068 raft_consensus.cc:2454] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 0.
I20240503 13:59:49.518633 1186 leader_election.cc:302] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ebad4bc2494af6a689f6d1bf05cc48, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.518780 1298 raft_consensus.cc:2781] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:49.518829 1298 raft_consensus.cc:482] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:49.518864 1298 raft_consensus.cc:3037] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.519917 964 catalog_manager.cc:5484] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176786c72902464ca12fbfb8c6a44767 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "176786c72902464ca12fbfb8c6a44767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:49.601742 1145 raft_consensus.cc:2454] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 1.
I20240503 13:59:49.601949 1143 raft_consensus.cc:2454] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 1.
I20240503 13:59:49.602804 1220 raft_consensus.cc:2454] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 1.
I20240503 13:59:49.603214 1032 leader_election.cc:302] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.603271 1275 raft_consensus.cc:504] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.601742 1217 raft_consensus.cc:2454] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ebad4bc2494af6a689f6d1bf05cc48 in term 1.
I20240503 13:59:49.603648 1219 raft_consensus.cc:2454] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176786c72902464ca12fbfb8c6a44767 in term 1.
I20240503 13:59:49.601742 1069 raft_consensus.cc:2454] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 1.
I20240503 13:59:49.603742 1275 leader_election.cc:288] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [CANDIDATE]: Term 1 election: Requested vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), afbf7d3c38d34497b671d263eef45878 (127.0.99.3:46843)
I20240503 13:59:49.604203 1184 leader_election.cc:302] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.604421 1270 raft_consensus.cc:2781] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.604512 1298 raft_consensus.cc:504] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.604648 1270 raft_consensus.cc:686] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 1 LEADER]: Becoming Leader. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Running, Role: LEADER
I20240503 13:59:49.604764 1306 raft_consensus.cc:2781] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.604792 1270 consensus_queue.cc:237] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.605135 1068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c1cd1783c14c1e9cd7deb2512e6584" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767"
I20240503 13:59:49.605270 1068 raft_consensus.cc:3037] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.605304 1306 raft_consensus.cc:686] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 1 LEADER]: Becoming Leader. State: Replica: 176786c72902464ca12fbfb8c6a44767, State: Running, Role: LEADER
I20240503 13:59:49.605513 1306 consensus_queue.cc:237] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.606348 1068 raft_consensus.cc:2454] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 1.
I20240503 13:59:49.606451 1219 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd7c505bbd54343bdcccc2b0f7d05ae" candidate_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afbf7d3c38d34497b671d263eef45878"
I20240503 13:59:49.606550 1219 raft_consensus.cc:3037] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.606678 1184 leader_election.cc:302] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.606882 1069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fd7c505bbd54343bdcccc2b0f7d05ae" candidate_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "176786c72902464ca12fbfb8c6a44767"
I20240503 13:59:49.606969 1069 raft_consensus.cc:3037] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.607177 964 catalog_manager.cc:5484] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 reported cstate change: term changed from 0 to 1, leader changed from <none> to afbf7d3c38d34497b671d263eef45878 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "afbf7d3c38d34497b671d263eef45878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:49.602804 1144 raft_consensus.cc:2454] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 1.
I20240503 13:59:49.607383 966 catalog_manager.cc:5484] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176786c72902464ca12fbfb8c6a44767 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "176786c72902464ca12fbfb8c6a44767" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:49.608049 1298 leader_election.cc:288] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [CANDIDATE]: Term 1 election: Requested vote from peers 176786c72902464ca12fbfb8c6a44767 (127.0.99.1:33661), 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2:46427)
I20240503 13:59:49.608098 1144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4c1cd1783c14c1e9cd7deb2512e6584" candidate_uuid: "afbf7d3c38d34497b671d263eef45878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ebad4bc2494af6a689f6d1bf05cc48"
I20240503 13:59:49.608146 1298 raft_consensus.cc:2781] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.608196 1144 raft_consensus.cc:3037] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:49.608222 1298 raft_consensus.cc:686] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 1 LEADER]: Becoming Leader. State: Replica: afbf7d3c38d34497b671d263eef45878, State: Running, Role: LEADER
I20240503 13:59:49.608340 1298 consensus_queue.cc:237] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.610844 964 catalog_manager.cc:5484] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 reported cstate change: term changed from 0 to 1, leader changed from <none> to afbf7d3c38d34497b671d263eef45878 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "afbf7d3c38d34497b671d263eef45878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:49.697423 1144 raft_consensus.cc:2454] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afbf7d3c38d34497b671d263eef45878 in term 1.
I20240503 13:59:49.697423 1219 raft_consensus.cc:2454] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ebad4bc2494af6a689f6d1bf05cc48 in term 1.
I20240503 13:59:49.697423 1069 raft_consensus.cc:2454] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ebad4bc2494af6a689f6d1bf05cc48 in term 1.
I20240503 13:59:49.698076 1109 leader_election.cc:302] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 176786c72902464ca12fbfb8c6a44767, 68ebad4bc2494af6a689f6d1bf05cc48, afbf7d3c38d34497b671d263eef45878; no voters:
I20240503 13:59:49.698356 1275 raft_consensus.cc:2781] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:49.698477 1275 raft_consensus.cc:686] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 LEADER]: Becoming Leader. State: Replica: 68ebad4bc2494af6a689f6d1bf05cc48, State: Running, Role: LEADER
I20240503 13:59:49.698642 1275 consensus_queue.cc:237] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } }
I20240503 13:59:49.701000 964 catalog_manager.cc:5484] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68ebad4bc2494af6a689f6d1bf05cc48 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:49.714715 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:49.717839 1319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:49.718289 1316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:49.719157 1317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:49.720031 396 server_base.cc:1034] running on GCE node
I20240503 13:59:49.720281 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:49.720336 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:49.720364 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744789720364 us; error 0 us; skew 500 ppm
I20240503 13:59:49.721685 396 webserver.cc:460] Webserver started at http://127.0.99.4:39985/ using document root <none> and password file <none>
I20240503 13:59:49.721874 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:49.721946 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:49.722026 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:49.722501 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-3-root/instance:
uuid: "158cf283538c4318996597f7de10d832"
format_stamp: "Formatted at 2024-05-03 13:59:49 on dist-test-slave-6w8v"
I20240503 13:59:49.724321 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240503 13:59:49.725862 1324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:49.726176 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:49.726258 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-3-root
uuid: "158cf283538c4318996597f7de10d832"
format_stamp: "Formatted at 2024-05-03 13:59:49 on dist-test-slave-6w8v"
I20240503 13:59:49.726352 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714744785267298-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:49.736147 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:49.736711 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:49.737053 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:49.737645 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:49.737695 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:49.737737 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:49.737767 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:49.744175 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:34843
I20240503 13:59:49.744205 1387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:34843 every 8 connection(s)
I20240503 13:59:49.750524 1388 heartbeater.cc:347] Connected to a master server at 127.0.99.62:46765
I20240503 13:59:49.750702 1388 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:49.751029 1388 heartbeater.cc:511] Master 127.0.99.62:46765 requested a full tablet report, sending...
I20240503 13:59:49.751899 964 ts_manager.cc:194] Registered new tserver with Master: 158cf283538c4318996597f7de10d832 (127.0.99.4:34843)
I20240503 13:59:49.752552 964 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50850
I20240503 13:59:49.843295 1298 consensus_queue.cc:1035] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.846814 1298 consensus_queue.cc:1035] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.905735 1306 consensus_queue.cc:1035] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.909718 1306 consensus_queue.cc:1035] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.916183 1303 consensus_queue.cc:1035] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.919091 1281 consensus_queue.cc:1035] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.983945 1270 consensus_queue.cc:1035] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:49.987308 1311 consensus_queue.cc:1035] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.047772 1281 consensus_queue.cc:1035] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.049626 1020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 13:59:50.051713 1303 consensus_queue.cc:1035] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.051818 1020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 13:59:50.052306 1020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 13:59:50.052366 1273 consensus_queue.cc:1035] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.055945 1406 consensus_queue.cc:1035] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.108719 1311 consensus_queue.cc:1035] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.112345 1298 consensus_queue.cc:1035] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.120869 1406 consensus_queue.cc:1035] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.124215 1406 consensus_queue.cc:1035] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:50.753732 1388 heartbeater.cc:503] Master 127.0.99.62:46765 was elected leader, sending a full tablet report...
I20240503 13:59:52.052781 1020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 13:59:52.053665 1020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 13:59:52.053769 1020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 13:59:54.054134 1020 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 13:59:54.054980 1020 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 13:59:54.055071 1020 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 13:59:55.064246 1219 consensus_queue.cc:237] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } attrs { replace: true } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } }
I20240503 13:59:55.066069 1144 raft_consensus.cc:1261] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Refusing update from remote peer afbf7d3c38d34497b671d263eef45878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.066069 1069 raft_consensus.cc:1261] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Refusing update from remote peer afbf7d3c38d34497b671d263eef45878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.066540 1311 consensus_queue.cc:1035] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.066789 1418 consensus_queue.cc:1035] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.068868 1419 raft_consensus.cc:2932] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } attrs { replace: true } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.069226 1069 raft_consensus.cc:2932] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } attrs { replace: true } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
W20240503 13:59:55.071807 1185 consensus_peers.cc:485] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 -> Peer 158cf283538c4318996597f7de10d832 (127.0.99.4:34843): Couldn't send request to peer 158cf283538c4318996597f7de10d832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a0e51b991cd4846920981e8274ff08b. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:55.071767 965 catalog_manager.cc:5484] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 reported cstate change: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "afbf7d3c38d34497b671d263eef45878" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.071873 1144 raft_consensus.cc:2932] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } attrs { replace: true } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.073704 1219 consensus_queue.cc:237] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } }
I20240503 13:59:55.075829 1069 raft_consensus.cc:1261] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Refusing update from remote peer afbf7d3c38d34497b671d263eef45878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:55.075881 1185 consensus_peers.cc:485] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 -> Peer 158cf283538c4318996597f7de10d832 (127.0.99.4:34843): Couldn't send request to peer 158cf283538c4318996597f7de10d832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8380321f903c4ffaa86ba33d02c09906. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:55.076264 1311 consensus_queue.cc:1035] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.076375 1144 raft_consensus.cc:1261] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Refusing update from remote peer afbf7d3c38d34497b671d263eef45878: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.076727 1419 consensus_queue.cc:1035] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.077965 1311 raft_consensus.cc:2932] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.078418 1069 raft_consensus.cc:2932] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.078514 1144 raft_consensus.cc:2932] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.080829 964 catalog_manager.cc:5484] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 reported cstate change: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "afbf7d3c38d34497b671d263eef45878" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.086421 1069 consensus_queue.cc:237] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } attrs { replace: true } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } }
I20240503 13:59:55.088263 1144 raft_consensus.cc:1261] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Refusing update from remote peer 176786c72902464ca12fbfb8c6a44767: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.088794 1219 raft_consensus.cc:1261] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Refusing update from remote peer 176786c72902464ca12fbfb8c6a44767: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.088873 1434 consensus_queue.cc:1035] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.089224 1281 consensus_queue.cc:1035] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.091154 1434 raft_consensus.cc:2932] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } attrs { replace: true } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.091648 1219 raft_consensus.cc:2932] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } attrs { replace: true } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.091666 1144 raft_consensus.cc:2932] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } attrs { replace: true } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.094363 966 catalog_manager.cc:5484] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 reported cstate change: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "176786c72902464ca12fbfb8c6a44767" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 13:59:55.095124 1034 consensus_peers.cc:485] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 -> Peer 158cf283538c4318996597f7de10d832 (127.0.99.4:34843): Couldn't send request to peer 158cf283538c4318996597f7de10d832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df636cac17de48aba6a1f0b2927d6e6d. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:55.095981 1069 consensus_queue.cc:237] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } }
I20240503 13:59:55.097710 1219 raft_consensus.cc:1261] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Refusing update from remote peer 176786c72902464ca12fbfb8c6a44767: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 13:59:55.098132 1034 consensus_peers.cc:485] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 -> Peer 158cf283538c4318996597f7de10d832 (127.0.99.4:34843): Couldn't send request to peer 158cf283538c4318996597f7de10d832. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9532ea79b8e4d518c3ea78908ed0765. This is attempt 1: this message will repeat every 5th retry.
I20240503 13:59:55.098438 1144 raft_consensus.cc:1261] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Refusing update from remote peer 176786c72902464ca12fbfb8c6a44767: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.098479 1434 consensus_queue.cc:1035] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.098783 1281 consensus_queue.cc:1035] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 13:59:55.100526 1434 raft_consensus.cc:2932] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.100960 1144 raft_consensus.cc:2932] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.100960 1219 raft_consensus.cc:2932] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } } }
I20240503 13:59:55.103448 965 catalog_manager.cc:5484] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 reported cstate change: config changed from index -1 to 2, NON_VOTER 158cf283538c4318996597f7de10d832 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "176786c72902464ca12fbfb8c6a44767" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "68ebad4bc2494af6a689f6d1bf05cc48" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 46427 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afbf7d3c38d34497b671d263eef45878" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 46843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "176786c72902464ca12fbfb8c6a44767" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 33661 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "158cf283538c4318996597f7de10d832" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34843 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 13:59:55.266490 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
W20240503 13:59:55.268429 951 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20240503 13:59:55.269465 1014 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 13:59:55.269588 1014 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 13:59:55.272387 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:55.272678 396 tablet_replica.cc:330] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767: stopping tablet replica
I20240503 13:59:55.272843 396 raft_consensus.cc:2229] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.273082 396 raft_consensus.cc:2258] T 8380321f903c4ffaa86ba33d02c09906 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 13:59:55.273082 1014 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 13:59:55.273183 1014 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 13:59:55.273856 396 tablet_replica.cc:330] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767: stopping tablet replica
I20240503 13:59:55.273969 396 raft_consensus.cc:2229] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.274092 396 raft_consensus.cc:2258] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.274523 396 tablet_replica.cc:330] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767: stopping tablet replica
I20240503 13:59:55.274639 396 raft_consensus.cc:2229] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.274744 396 raft_consensus.cc:2258] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.275171 396 tablet_replica.cc:330] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767: stopping tablet replica
I20240503 13:59:55.275280 396 raft_consensus.cc:2229] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.275380 396 raft_consensus.cc:2258] T f4c1cd1783c14c1e9cd7deb2512e6584 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.275777 396 tablet_replica.cc:330] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767: stopping tablet replica
I20240503 13:59:55.275887 396 raft_consensus.cc:2229] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.276161 396 raft_consensus.cc:2258] T c9532ea79b8e4d518c3ea78908ed0765 P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.276786 396 tablet_replica.cc:330] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767: stopping tablet replica
I20240503 13:59:55.276898 396 raft_consensus.cc:2229] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.276996 396 raft_consensus.cc:2258] T 6a0e51b991cd4846920981e8274ff08b P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.277469 396 tablet_replica.cc:330] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767: stopping tablet replica
I20240503 13:59:55.277559 396 raft_consensus.cc:2229] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.277753 396 raft_consensus.cc:2258] T df636cac17de48aba6a1f0b2927d6e6d P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.278209 396 tablet_replica.cc:330] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767: stopping tablet replica
I20240503 13:59:55.278295 396 raft_consensus.cc:2229] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.278455 396 raft_consensus.cc:2258] T e4484a16802b4141a38cd0008bbddecc P 176786c72902464ca12fbfb8c6a44767 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.293671 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 13:59:55.296893 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 13:59:55.302858 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:55.303030 396 tablet_replica.cc:330] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48: stopping tablet replica
I20240503 13:59:55.303185 396 raft_consensus.cc:2229] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.303326 396 raft_consensus.cc:2258] T e4484a16802b4141a38cd0008bbddecc P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.303824 396 tablet_replica.cc:330] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48: stopping tablet replica
I20240503 13:59:55.303936 396 raft_consensus.cc:2229] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.304054 396 raft_consensus.cc:2258] T df636cac17de48aba6a1f0b2927d6e6d P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.304600 396 tablet_replica.cc:330] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48: stopping tablet replica
I20240503 13:59:55.304697 396 raft_consensus.cc:2229] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.304800 396 raft_consensus.cc:2258] T f4c1cd1783c14c1e9cd7deb2512e6584 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.305140 396 tablet_replica.cc:330] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48: stopping tablet replica
I20240503 13:59:55.305241 396 raft_consensus.cc:2229] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.305418 396 raft_consensus.cc:2258] T 3fd7c505bbd54343bdcccc2b0f7d05ae P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.305788 396 tablet_replica.cc:330] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48: stopping tablet replica
I20240503 13:59:55.305891 396 raft_consensus.cc:2229] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.306015 396 raft_consensus.cc:2258] T 8380321f903c4ffaa86ba33d02c09906 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.306504 396 tablet_replica.cc:330] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48: stopping tablet replica
I20240503 13:59:55.306605 396 raft_consensus.cc:2229] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.306794 396 raft_consensus.cc:2258] T 06c5d53b0cf445c19ca0a49da5e70fb0 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.307101 396 tablet_replica.cc:330] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48: stopping tablet replica
I20240503 13:59:55.307179 396 raft_consensus.cc:2229] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.307307 396 raft_consensus.cc:2258] T c9532ea79b8e4d518c3ea78908ed0765 P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.307809 396 tablet_replica.cc:330] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48: stopping tablet replica
I20240503 13:59:55.307905 396 raft_consensus.cc:2229] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.308010 396 raft_consensus.cc:2258] T 6a0e51b991cd4846920981e8274ff08b P 68ebad4bc2494af6a689f6d1bf05cc48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.311741 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 13:59:55.314641 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
W20240503 13:59:55.316061 1014 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 13:59:55.316169 1014 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 13:59:55.317919 1014 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240503 13:59:55.317992 1014 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240503 13:59:55.319614 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:55.319783 396 tablet_replica.cc:330] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878: stopping tablet replica
I20240503 13:59:55.319903 396 raft_consensus.cc:2229] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.320013 396 raft_consensus.cc:2258] T 3fd7c505bbd54343bdcccc2b0f7d05ae P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.320425 396 tablet_replica.cc:330] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878: stopping tablet replica
I20240503 13:59:55.320518 396 raft_consensus.cc:2229] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.320683 396 raft_consensus.cc:2258] T f4c1cd1783c14c1e9cd7deb2512e6584 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.320986 396 tablet_replica.cc:330] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878: stopping tablet replica
I20240503 13:59:55.321065 396 raft_consensus.cc:2229] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.321135 396 raft_consensus.cc:2258] T e4484a16802b4141a38cd0008bbddecc P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.321406 396 tablet_replica.cc:330] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878: stopping tablet replica
I20240503 13:59:55.321480 396 raft_consensus.cc:2229] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.321556 396 raft_consensus.cc:2258] T c9532ea79b8e4d518c3ea78908ed0765 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.322018 396 tablet_replica.cc:330] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878: stopping tablet replica
I20240503 13:59:55.322099 396 raft_consensus.cc:2229] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.322261 396 raft_consensus.cc:2258] T 6a0e51b991cd4846920981e8274ff08b P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.322681 396 tablet_replica.cc:330] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878: stopping tablet replica
I20240503 13:59:55.322760 396 raft_consensus.cc:2229] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.322837 396 raft_consensus.cc:2258] T df636cac17de48aba6a1f0b2927d6e6d P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.323235 396 tablet_replica.cc:330] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878: stopping tablet replica
I20240503 13:59:55.323307 396 raft_consensus.cc:2229] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:55.323376 396 raft_consensus.cc:2258] T 06c5d53b0cf445c19ca0a49da5e70fb0 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.323689 396 tablet_replica.cc:330] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878: stopping tablet replica
I20240503 13:59:55.323758 396 raft_consensus.cc:2229] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.323922 396 raft_consensus.cc:2258] T 8380321f903c4ffaa86ba33d02c09906 P afbf7d3c38d34497b671d263eef45878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.327919 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 13:59:55.330507 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 13:59:55.334247 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:55.346599 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 13:59:55.348888 396 master.cc:537] Master@127.0.99.62:46765 shutting down...
I20240503 13:59:55.352347 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:55.352537 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:55.352613 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P e77e718e8c6046cc9b6f849f4622706f: stopping tablet replica
I20240503 13:59:55.356114 396 master.cc:552] Master@127.0.99.62:46765 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7350 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240503 13:59:55.365731 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:40969,127.0.99.61:37451,127.0.99.60:46791
I20240503 13:59:55.366168 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:55.368793 1443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.369652 1442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.370158 1445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.370620 396 server_base.cc:1034] running on GCE node
I20240503 13:59:55.370821 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:55.370859 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:55.370873 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744795370873 us; error 0 us; skew 500 ppm
I20240503 13:59:55.371930 396 webserver.cc:460] Webserver started at http://127.0.99.62:35137/ using document root <none> and password file <none>
I20240503 13:59:55.372072 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:55.372112 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:55.372166 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:55.372532 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "4e3386fa11264bb8af670c04a9a3c472"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.373991 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:55.374868 1450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.375130 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:55.375217 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "4e3386fa11264bb8af670c04a9a3c472"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.375285 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:55.386451 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:55.386827 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:55.390743 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:40969
I20240503 13:59:55.391956 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:55.396180 1502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:40969 every 8 connection(s)
I20240503 13:59:55.396675 1503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.397840 1503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
W20240503 13:59:55.398192 1508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.398653 1506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.399708 1505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.400399 396 server_base.cc:1034] running on GCE node
I20240503 13:59:55.400537 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:55.400575 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:55.400594 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744795400594 us; error 0 us; skew 500 ppm
I20240503 13:59:55.401553 396 webserver.cc:460] Webserver started at http://127.0.99.61:45451/ using document root <none> and password file <none>
I20240503 13:59:55.401716 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:55.401765 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:55.401837 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:55.402257 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root/instance:
uuid: "39c96fed49e74c7fa7eb67711b7a131b"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.402509 1503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } has no permanent_uuid. Determining permanent_uuid...
W20240503 13:59:55.403821 1503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.61:37451: Network error: Client connection negotiation failed: client connection to 127.0.99.61:37451: connect: Connection refused (error 111)
I20240503 13:59:55.403955 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:55.404878 1516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.405143 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:55.405203 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root
uuid: "39c96fed49e74c7fa7eb67711b7a131b"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.405256 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:55.427881 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:55.428273 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:55.432121 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.61:37451
I20240503 13:59:55.432152 1568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.61:37451 every 8 connection(s)
I20240503 13:59:55.432886 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:55.433049 1569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.434577 1569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
W20240503 13:59:55.435258 1571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.435786 1574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.436586 1572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.437312 396 server_base.cc:1034] running on GCE node
I20240503 13:59:55.437471 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:55.437507 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:55.437534 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744795437533 us; error 0 us; skew 500 ppm
I20240503 13:59:55.438529 396 webserver.cc:460] Webserver started at http://127.0.99.60:36727/ using document root <none> and password file <none>
I20240503 13:59:55.438699 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:55.438750 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:55.438818 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:55.438977 1569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:55.439230 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root/instance:
uuid: "9c8f54d996104f10a8c8e158be67adf4"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.440935 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:55.442060 1581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.442325 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:55.442404 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root
uuid: "9c8f54d996104f10a8c8e158be67adf4"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.442478 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:55.443202 1569 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
W20240503 13:59:55.444397 1569 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:46791: Network error: Client connection negotiation failed: client connection to 127.0.99.60:46791: connect: Connection refused (error 111)
I20240503 13:59:55.451982 1503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } attempt: 1
I20240503 13:59:55.456194 1503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
W20240503 13:59:55.457350 1503 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.0.99.60:46791: Network error: Client connection negotiation failed: client connection to 127.0.99.60:46791: connect: Connection refused (error 111)
I20240503 13:59:55.463069 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:55.463446 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:55.467446 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.60:46791
I20240503 13:59:55.467484 1634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.60:46791 every 8 connection(s)
I20240503 13:59:55.467941 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 13:59:55.468386 1635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:55.469512 1635 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:55.473555 1635 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:55.477156 1635 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } has no permanent_uuid. Determining permanent_uuid...
I20240503 13:59:55.478549 1569 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } attempt: 1
I20240503 13:59:55.482374 1635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: Bootstrap starting.
I20240503 13:59:55.482828 1569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b: Bootstrap starting.
I20240503 13:59:55.483264 1635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:55.483647 1569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:55.484428 1635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: No bootstrap required, opened a new log
I20240503 13:59:55.484617 1569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b: No bootstrap required, opened a new log
I20240503 13:59:55.484869 1635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.484973 1635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:55.485019 1635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8f54d996104f10a8c8e158be67adf4, State: Initialized, Role: FOLLOWER
I20240503 13:59:55.484982 1569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.485074 1569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:55.485114 1569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c96fed49e74c7fa7eb67711b7a131b, State: Initialized, Role: FOLLOWER
I20240503 13:59:55.485154 1635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.485239 1569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.485453 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.485541 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:55.485517 1643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.485603 1643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:55.485626 1635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:55.485670 1569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [sys.catalog]: configured and running, proceeding with master startup.
W20240503 13:59:55.487810 1664 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 13:59:55.487811 1665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 13:59:55.487888 1664 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 13:59:55.487916 1665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240503 13:59:55.496533 1503 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } attempt: 1
I20240503 13:59:55.501693 1503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472: Bootstrap starting.
I20240503 13:59:55.502698 1503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:55.503937 1503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472: No bootstrap required, opened a new log
I20240503 13:59:55.504405 1503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.504525 1503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:55.504555 1503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3386fa11264bb8af670c04a9a3c472, State: Initialized, Role: FOLLOWER
I20240503 13:59:55.504688 1503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.505052 1667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.505163 1667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:55.505200 1503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:55.506994 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240503 13:59:55.507062 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240503 13:59:55.507553 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:55.508205 1678 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240503 13:59:55.508289 1678 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240503 13:59:55.510215 1679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.510658 1682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.511608 396 server_base.cc:1034] running on GCE node
W20240503 13:59:55.512236 1680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.512482 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:55.512527 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:55.512548 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744795512548 us; error 0 us; skew 500 ppm
I20240503 13:59:55.513554 396 webserver.cc:460] Webserver started at http://127.0.99.1:34235/ using document root <none> and password file <none>
I20240503 13:59:55.513710 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:55.513774 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:55.513857 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:55.514299 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root/instance:
uuid: "8e2068339c104900a488c7d91231caae"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.515796 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:55.516737 1687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.516970 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:55.517038 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root
uuid: "8e2068339c104900a488c7d91231caae"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.517112 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:55.540263 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:55.540726 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:55.541036 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:55.541520 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:55.541566 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.541610 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:55.541637 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.546428 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:35443
I20240503 13:59:55.546460 1750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:35443 every 8 connection(s)
I20240503 13:59:55.547470 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:55.550750 1761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.551580 1759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.552031 1758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.552368 1752 heartbeater.cc:347] Connected to a master server at 127.0.99.62:40969
I20240503 13:59:55.552394 1751 heartbeater.cc:347] Connected to a master server at 127.0.99.60:46791
I20240503 13:59:55.552481 1751 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:55.552490 396 server_base.cc:1034] running on GCE node
I20240503 13:59:55.552469 1752 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:55.552767 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 13:59:55.552778 1753 heartbeater.cc:347] Connected to a master server at 127.0.99.61:37451
I20240503 13:59:55.552840 1752 heartbeater.cc:511] Master 127.0.99.62:40969 requested a full tablet report, sending...
I20240503 13:59:55.552846 1751 heartbeater.cc:511] Master 127.0.99.60:46791 requested a full tablet report, sending...
I20240503 13:59:55.552881 1753 heartbeater.cc:464] Registering TS with master...
W20240503 13:59:55.552822 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:55.553082 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744795553079 us; error 0 us; skew 500 ppm
I20240503 13:59:55.553217 1753 heartbeater.cc:511] Master 127.0.99.61:37451 requested a full tablet report, sending...
I20240503 13:59:55.553630 1599 ts_manager.cc:194] Registered new tserver with Master: 8e2068339c104900a488c7d91231caae (127.0.99.1:35443)
I20240503 13:59:55.553627 1467 ts_manager.cc:194] Registered new tserver with Master: 8e2068339c104900a488c7d91231caae (127.0.99.1:35443)
I20240503 13:59:55.554121 1533 ts_manager.cc:194] Registered new tserver with Master: 8e2068339c104900a488c7d91231caae (127.0.99.1:35443)
I20240503 13:59:55.554255 396 webserver.cc:460] Webserver started at http://127.0.99.2:33943/ using document root <none> and password file <none>
I20240503 13:59:55.554412 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:55.554464 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:55.554548 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:55.554931 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root/instance:
uuid: "b4d222fec4c3442eab3d2ae568949686"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.556391 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:55.557278 1766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.557533 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:55.557601 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root
uuid: "b4d222fec4c3442eab3d2ae568949686"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.557667 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:55.575448 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:55.575932 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:55.576413 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:55.576936 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:55.576977 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.577020 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:55.577044 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.581542 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:40189
I20240503 13:59:55.581583 1829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:40189 every 8 connection(s)
I20240503 13:59:55.582545 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 13:59:55.585860 396 server_base.cc:1034] running on GCE node
W20240503 13:59:55.586019 1840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:55.586539 1837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.587162 1832 heartbeater.cc:347] Connected to a master server at 127.0.99.61:37451
I20240503 13:59:55.587260 1832 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:55.587288 1830 heartbeater.cc:347] Connected to a master server at 127.0.99.60:46791
I20240503 13:59:55.587383 1830 heartbeater.cc:464] Registering TS with master...
W20240503 13:59:55.587481 1838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:55.587610 1830 heartbeater.cc:511] Master 127.0.99.60:46791 requested a full tablet report, sending...
I20240503 13:59:55.587523 1832 heartbeater.cc:511] Master 127.0.99.61:37451 requested a full tablet report, sending...
I20240503 13:59:55.587849 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 13:59:55.587858 1831 heartbeater.cc:347] Connected to a master server at 127.0.99.62:40969
W20240503 13:59:55.587908 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:55.587932 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744795587931 us; error 0 us; skew 500 ppm
I20240503 13:59:55.587934 1831 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:55.588146 1831 heartbeater.cc:511] Master 127.0.99.62:40969 requested a full tablet report, sending...
I20240503 13:59:55.588438 1533 ts_manager.cc:194] Registered new tserver with Master: b4d222fec4c3442eab3d2ae568949686 (127.0.99.2:40189)
I20240503 13:59:55.588536 1599 ts_manager.cc:194] Registered new tserver with Master: b4d222fec4c3442eab3d2ae568949686 (127.0.99.2:40189)
I20240503 13:59:55.588963 1467 ts_manager.cc:194] Registered new tserver with Master: b4d222fec4c3442eab3d2ae568949686 (127.0.99.2:40189)
I20240503 13:59:55.589183 396 webserver.cc:460] Webserver started at http://127.0.99.3:38313/ using document root <none> and password file <none>
I20240503 13:59:55.589339 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:55.589390 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:55.589460 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:55.589836 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root/instance:
uuid: "b1914f7013d7408dbfd70691446382fb"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.591272 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:55.592361 1845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.592622 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:55.592689 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root
uuid: "b1914f7013d7408dbfd70691446382fb"
format_stamp: "Formatted at 2024-05-03 13:59:55 on dist-test-slave-6w8v"
I20240503 13:59:55.592751 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714744785267298-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:55.611582 1667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 13:59:55.611764 1667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.612241 1667 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39c96fed49e74c7fa7eb67711b7a131b (127.0.99.61:37451), 9c8f54d996104f10a8c8e158be67adf4 (127.0.99.60:46791)
I20240503 13:59:55.612694 1609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e3386fa11264bb8af670c04a9a3c472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c8f54d996104f10a8c8e158be67adf4" is_pre_election: true
I20240503 13:59:55.612681 1543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e3386fa11264bb8af670c04a9a3c472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c96fed49e74c7fa7eb67711b7a131b" is_pre_election: true
I20240503 13:59:55.612869 1609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e3386fa11264bb8af670c04a9a3c472 in term 0.
I20240503 13:59:55.612869 1543 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e3386fa11264bb8af670c04a9a3c472 in term 0.
I20240503 13:59:55.613215 1451 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c96fed49e74c7fa7eb67711b7a131b, 4e3386fa11264bb8af670c04a9a3c472; no voters:
I20240503 13:59:55.613377 1667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 13:59:55.613428 1667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 13:59:55.613464 1667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:55.614342 1667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.614534 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:55.614718 1667 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [CANDIDATE]: Term 1 election: Requested vote from peers 39c96fed49e74c7fa7eb67711b7a131b (127.0.99.61:37451), 9c8f54d996104f10a8c8e158be67adf4 (127.0.99.60:46791)
I20240503 13:59:55.614928 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:55.615029 1543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e3386fa11264bb8af670c04a9a3c472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c96fed49e74c7fa7eb67711b7a131b"
I20240503 13:59:55.615167 1543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:55.615286 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:55.615670 1609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4e3386fa11264bb8af670c04a9a3c472" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c8f54d996104f10a8c8e158be67adf4"
I20240503 13:59:55.615787 1609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:55.615825 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:55.615864 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.615906 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:55.615923 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:55.616287 1543 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e3386fa11264bb8af670c04a9a3c472 in term 1.
I20240503 13:59:55.616575 1609 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e3386fa11264bb8af670c04a9a3c472 in term 1.
I20240503 13:59:55.616581 1451 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c96fed49e74c7fa7eb67711b7a131b, 4e3386fa11264bb8af670c04a9a3c472; no voters:
I20240503 13:59:55.616763 1667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:55.616966 1667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3386fa11264bb8af670c04a9a3c472, State: Running, Role: LEADER
I20240503 13:59:55.617157 1667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:55.617888 1853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3386fa11264bb8af670c04a9a3c472. Latest consensus state: current_term: 1 leader_uuid: "4e3386fa11264bb8af670c04a9a3c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.618005 1853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:55.618270 1866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:55.619226 1866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:55.621248 1609 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 1 FOLLOWER]: Refusing update from remote peer 4e3386fa11264bb8af670c04a9a3c472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.621218 1543 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 1 FOLLOWER]: Refusing update from remote peer 4e3386fa11264bb8af670c04a9a3c472: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 13:59:55.621235 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:42055
I20240503 13:59:55.621484 1912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:42055 every 8 connection(s)
I20240503 13:59:55.621627 1853 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:55.621822 1667 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 13:59:55.623066 1643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3386fa11264bb8af670c04a9a3c472. Latest consensus state: current_term: 1 leader_uuid: "4e3386fa11264bb8af670c04a9a3c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.623183 1643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:55.623142 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3386fa11264bb8af670c04a9a3c472. Latest consensus state: current_term: 1 leader_uuid: "4e3386fa11264bb8af670c04a9a3c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.623234 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:55.624272 1667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e3386fa11264bb8af670c04a9a3c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.624384 1667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:55.624632 1642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e3386fa11264bb8af670c04a9a3c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.624727 1642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:55.624680 1866 catalog_manager.cc:1347] Generated new cluster ID: e9cab69efc68448ba90a25b93acd3b93
I20240503 13:59:55.624984 1866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:55.628628 1915 heartbeater.cc:347] Connected to a master server at 127.0.99.61:37451
I20240503 13:59:55.628728 1915 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:55.628984 1915 heartbeater.cc:511] Master 127.0.99.61:37451 requested a full tablet report, sending...
I20240503 13:59:55.629588 1667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4e3386fa11264bb8af670c04a9a3c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.629704 1667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:55.630549 1533 ts_manager.cc:194] Registered new tserver with Master: b1914f7013d7408dbfd70691446382fb (127.0.99.3:42055)
I20240503 13:59:55.630511 1643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4e3386fa11264bb8af670c04a9a3c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:55.630719 1643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [sys.catalog]: This master's current role is: FOLLOWER
I20240503 13:59:55.631790 1914 heartbeater.cc:347] Connected to a master server at 127.0.99.62:40969
I20240503 13:59:55.631875 1914 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:55.632090 1914 heartbeater.cc:511] Master 127.0.99.62:40969 requested a full tablet report, sending...
I20240503 13:59:55.632588 1913 heartbeater.cc:347] Connected to a master server at 127.0.99.60:46791
I20240503 13:59:55.632665 1913 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:55.632715 1467 ts_manager.cc:194] Registered new tserver with Master: b1914f7013d7408dbfd70691446382fb (127.0.99.3:42055)
I20240503 13:59:55.632845 1913 heartbeater.cc:511] Master 127.0.99.60:46791 requested a full tablet report, sending...
I20240503 13:59:55.634851 1599 ts_manager.cc:194] Registered new tserver with Master: b1914f7013d7408dbfd70691446382fb (127.0.99.3:42055)
I20240503 13:59:55.635759 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013790296s
I20240503 13:59:55.636010 1866 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:55.636745 1866 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:55.642627 1866 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472: Generated new TSK 0
I20240503 13:59:55.642832 1866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:56.488790 1665 catalog_manager.cc:1259] Loaded cluster ID: e9cab69efc68448ba90a25b93acd3b93
I20240503 13:59:56.488790 1664 catalog_manager.cc:1259] Loaded cluster ID: e9cab69efc68448ba90a25b93acd3b93
I20240503 13:59:56.488917 1665 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b: loading cluster ID for follower catalog manager: success
I20240503 13:59:56.488938 1664 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: loading cluster ID for follower catalog manager: success
I20240503 13:59:56.490262 1665 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b: acquiring CA information for follower catalog manager: success
I20240503 13:59:56.490263 1664 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: acquiring CA information for follower catalog manager: success
I20240503 13:59:56.490972 1664 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 13:59:56.490972 1665 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240503 13:59:56.556324 1467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40720
I20240503 13:59:56.590543 1467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40736
I20240503 13:59:56.635046 1467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40746
I20240503 13:59:57.507167 1677 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 13:59:57.557325 1752 heartbeater.cc:503] Master 127.0.99.62:40969 was elected leader, sending a full tablet report...
I20240503 13:59:57.591460 1831 heartbeater.cc:503] Master 127.0.99.62:40969 was elected leader, sending a full tablet report...
I20240503 13:59:57.635983 1914 heartbeater.cc:503] Master 127.0.99.62:40969 was elected leader, sending a full tablet report...
I20240503 13:59:57.789083 396 test_util.cc:268] Using random seed: 579697557
I20240503 13:59:57.800925 1467 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40748:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 13:59:57.818218 1794 tablet_service.cc:1467] Processing CreateTablet for tablet 4873933fd77643a6bfb683dcaee20c81 (DEFAULT_TABLE table=test-workload [id=dfc7bdb5f0c04f5d914597850793cbba]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240503 13:59:57.818272 1868 tablet_service.cc:1467] Processing CreateTablet for tablet 730adc221d8b40e78f9884e21a6f569f (DEFAULT_TABLE table=test-workload [id=dfc7bdb5f0c04f5d914597850793cbba]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240503 13:59:57.818826 1794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4873933fd77643a6bfb683dcaee20c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.819048 1868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730adc221d8b40e78f9884e21a6f569f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.819784 1715 tablet_service.cc:1467] Processing CreateTablet for tablet 87b96db84666482d984a0fbe1f6d19da (DEFAULT_TABLE table=test-workload [id=dfc7bdb5f0c04f5d914597850793cbba]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240503 13:59:57.820284 1715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87b96db84666482d984a0fbe1f6d19da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:57.821504 1956 tablet_bootstrap.cc:492] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686: Bootstrap starting.
I20240503 13:59:57.822712 1956 tablet_bootstrap.cc:654] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:57.823483 1958 tablet_bootstrap.cc:492] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae: Bootstrap starting.
I20240503 13:59:57.824040 1956 tablet_bootstrap.cc:492] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686: No bootstrap required, opened a new log
I20240503 13:59:57.824059 1957 tablet_bootstrap.cc:492] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb: Bootstrap starting.
I20240503 13:59:57.824188 1956 ts_tablet_manager.cc:1367] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 13:59:57.824681 1956 raft_consensus.cc:348] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d222fec4c3442eab3d2ae568949686" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40189 } }
I20240503 13:59:57.824806 1956 raft_consensus.cc:374] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:57.824841 1956 raft_consensus.cc:729] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d222fec4c3442eab3d2ae568949686, State: Initialized, Role: FOLLOWER
I20240503 13:59:57.824891 1958 tablet_bootstrap.cc:654] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:57.825042 1956 consensus_queue.cc:260] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d222fec4c3442eab3d2ae568949686" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40189 } }
I20240503 13:59:57.825093 1957 tablet_bootstrap.cc:654] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:57.825153 1956 raft_consensus.cc:388] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:57.825191 1956 raft_consensus.cc:482] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:57.825237 1956 raft_consensus.cc:3037] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:57.826380 1958 tablet_bootstrap.cc:492] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae: No bootstrap required, opened a new log
I20240503 13:59:57.826335 1956 raft_consensus.cc:504] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d222fec4c3442eab3d2ae568949686" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40189 } }
I20240503 13:59:57.826529 1958 ts_tablet_manager.cc:1367] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 13:59:57.826570 1956 leader_election.cc:302] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4d222fec4c3442eab3d2ae568949686; no voters:
I20240503 13:59:57.826930 1956 leader_election.cc:288] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:57.827026 1962 raft_consensus.cc:2781] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:57.827116 1958 raft_consensus.cc:348] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2068339c104900a488c7d91231caae" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35443 } }
I20240503 13:59:57.827199 1962 raft_consensus.cc:686] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 1 LEADER]: Becoming Leader. State: Replica: b4d222fec4c3442eab3d2ae568949686, State: Running, Role: LEADER
I20240503 13:59:57.827248 1958 raft_consensus.cc:374] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:57.827284 1958 raft_consensus.cc:729] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e2068339c104900a488c7d91231caae, State: Initialized, Role: FOLLOWER
I20240503 13:59:57.827309 1956 ts_tablet_manager.cc:1398] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 13:59:57.827339 1962 consensus_queue.cc:237] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d222fec4c3442eab3d2ae568949686" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40189 } }
I20240503 13:59:57.827442 1958 consensus_queue.cc:260] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2068339c104900a488c7d91231caae" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35443 } }
I20240503 13:59:57.827538 1958 raft_consensus.cc:388] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:57.827608 1958 raft_consensus.cc:482] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:57.827659 1958 raft_consensus.cc:3037] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:57.828785 1958 raft_consensus.cc:504] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2068339c104900a488c7d91231caae" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35443 } }
I20240503 13:59:57.828938 1958 leader_election.cc:302] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e2068339c104900a488c7d91231caae; no voters:
I20240503 13:59:57.828859 1467 catalog_manager.cc:5484] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4d222fec4c3442eab3d2ae568949686 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "b4d222fec4c3442eab3d2ae568949686" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d222fec4c3442eab3d2ae568949686" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40189 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:57.829232 1958 leader_election.cc:288] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:57.830559 1958 ts_tablet_manager.cc:1398] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240503 13:59:57.834853 1957 tablet_bootstrap.cc:492] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb: No bootstrap required, opened a new log
I20240503 13:59:57.834992 1957 ts_tablet_manager.cc:1367] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 13:59:57.835567 1957 raft_consensus.cc:348] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1914f7013d7408dbfd70691446382fb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42055 } }
I20240503 13:59:57.835748 1957 raft_consensus.cc:374] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:57.835781 1957 raft_consensus.cc:729] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1914f7013d7408dbfd70691446382fb, State: Initialized, Role: FOLLOWER
I20240503 13:59:57.835808 1967 raft_consensus.cc:2781] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:57.835898 1957 consensus_queue.cc:260] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1914f7013d7408dbfd70691446382fb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42055 } }
I20240503 13:59:57.836006 1957 raft_consensus.cc:388] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:57.836041 1957 raft_consensus.cc:482] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:57.836082 1957 raft_consensus.cc:3037] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:57.836102 1967 raft_consensus.cc:686] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 1 LEADER]: Becoming Leader. State: Replica: 8e2068339c104900a488c7d91231caae, State: Running, Role: LEADER
I20240503 13:59:57.836437 1967 consensus_queue.cc:237] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2068339c104900a488c7d91231caae" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35443 } }
I20240503 13:59:57.837456 1957 raft_consensus.cc:504] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1914f7013d7408dbfd70691446382fb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42055 } }
I20240503 13:59:57.838740 1467 catalog_manager.cc:5484] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e2068339c104900a488c7d91231caae (127.0.99.1). New cstate: current_term: 1 leader_uuid: "8e2068339c104900a488c7d91231caae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e2068339c104900a488c7d91231caae" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 35443 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:57.838997 1957 leader_election.cc:302] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1914f7013d7408dbfd70691446382fb; no voters:
I20240503 13:59:57.839224 1957 leader_election.cc:288] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:57.839637 1957 ts_tablet_manager.cc:1398] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240503 13:59:57.840287 1973 raft_consensus.cc:2781] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:57.840507 1973 raft_consensus.cc:686] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 1 LEADER]: Becoming Leader. State: Replica: b1914f7013d7408dbfd70691446382fb, State: Running, Role: LEADER
I20240503 13:59:57.841102 1973 consensus_queue.cc:237] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1914f7013d7408dbfd70691446382fb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42055 } }
I20240503 13:59:57.844655 1466 catalog_manager.cc:5484] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb reported cstate change: term changed from 0 to 1, leader changed from <none> to b1914f7013d7408dbfd70691446382fb (127.0.99.3). New cstate: current_term: 1 leader_uuid: "b1914f7013d7408dbfd70691446382fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1914f7013d7408dbfd70691446382fb" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 42055 } health_report { overall_health: HEALTHY } } }
I20240503 13:59:57.851425 396 master.cc:537] Master@127.0.99.62:40969 shutting down...
W20240503 13:59:57.852293 1914 heartbeater.cc:644] Failed to heartbeat to 127.0.99.62:40969 (0 consecutive failures): Remote error: Failed to send heartbeat to master: Service unavailable: service kudu.master.MasterService not registered on Master
I20240503 13:59:57.855228 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:57.855531 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:57.855660 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4e3386fa11264bb8af670c04a9a3c472: stopping tablet replica
I20240503 13:59:57.869809 396 master.cc:552] Master@127.0.99.62:40969 shutdown complete.
I20240503 13:59:59.351732 1980 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e3386fa11264bb8af670c04a9a3c472)
I20240503 13:59:59.351902 1980 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:59.352355 1980 leader_election.cc:288] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e3386fa11264bb8af670c04a9a3c472 (127.0.99.62:40969), 9c8f54d996104f10a8c8e158be67adf4 (127.0.99.60:46791)
I20240503 13:59:59.353106 1609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39c96fed49e74c7fa7eb67711b7a131b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9c8f54d996104f10a8c8e158be67adf4" is_pre_election: true
W20240503 13:59:59.353602 1520 leader_election.cc:334] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e3386fa11264bb8af670c04a9a3c472 (127.0.99.62:40969): Network error: Client connection negotiation failed: client connection to 127.0.99.62:40969: connect: Connection refused (error 111)
I20240503 13:59:59.353679 1520 leader_election.cc:302] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39c96fed49e74c7fa7eb67711b7a131b; no voters: 4e3386fa11264bb8af670c04a9a3c472, 9c8f54d996104f10a8c8e158be67adf4
I20240503 13:59:59.353828 1980 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 13:59:59.354795 1983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4e3386fa11264bb8af670c04a9a3c472)
I20240503 13:59:59.354895 1983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:59.355264 1983 leader_election.cc:288] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e3386fa11264bb8af670c04a9a3c472 (127.0.99.62:40969), 39c96fed49e74c7fa7eb67711b7a131b (127.0.99.61:37451)
I20240503 13:59:59.355674 1543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c8f54d996104f10a8c8e158be67adf4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "39c96fed49e74c7fa7eb67711b7a131b" is_pre_election: true
I20240503 13:59:59.355789 1543 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c8f54d996104f10a8c8e158be67adf4 in term 1.
I20240503 13:59:59.356073 1582 leader_election.cc:302] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c96fed49e74c7fa7eb67711b7a131b, 9c8f54d996104f10a8c8e158be67adf4; no voters:
I20240503 13:59:59.356222 1983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 13:59:59.356280 1983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4e3386fa11264bb8af670c04a9a3c472)
I20240503 13:59:59.356328 1983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:59:59.356421 1585 leader_election.cc:334] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4e3386fa11264bb8af670c04a9a3c472 (127.0.99.62:40969): Network error: Client connection negotiation failed: client connection to 127.0.99.62:40969: connect: Connection refused (error 111)
I20240503 13:59:59.359337 1983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:59.360004 1983 leader_election.cc:288] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [CANDIDATE]: Term 2 election: Requested vote from peers 4e3386fa11264bb8af670c04a9a3c472 (127.0.99.62:40969), 39c96fed49e74c7fa7eb67711b7a131b (127.0.99.61:37451)
I20240503 13:59:59.360414 1543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9c8f54d996104f10a8c8e158be67adf4" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "39c96fed49e74c7fa7eb67711b7a131b"
I20240503 13:59:59.360538 1543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 1 FOLLOWER]: Advancing to term 2
W20240503 13:59:59.361078 1585 leader_election.cc:334] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4e3386fa11264bb8af670c04a9a3c472 (127.0.99.62:40969): Network error: Client connection negotiation failed: client connection to 127.0.99.62:40969: connect: Connection refused (error 111)
I20240503 13:59:59.361418 1543 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c8f54d996104f10a8c8e158be67adf4 in term 2.
I20240503 13:59:59.361785 1582 leader_election.cc:302] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39c96fed49e74c7fa7eb67711b7a131b, 9c8f54d996104f10a8c8e158be67adf4; no voters: 4e3386fa11264bb8af670c04a9a3c472
I20240503 13:59:59.362021 1983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 13:59:59.362366 1983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 2 LEADER]: Becoming Leader. State: Replica: 9c8f54d996104f10a8c8e158be67adf4, State: Running, Role: LEADER
I20240503 13:59:59.362679 1983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } }
I20240503 13:59:59.363904 1986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c8f54d996104f10a8c8e158be67adf4. Latest consensus state: current_term: 2 leader_uuid: "9c8f54d996104f10a8c8e158be67adf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e3386fa11264bb8af670c04a9a3c472" member_type: VOTER last_known_addr { host: "127.0.99.62" port: 40969 } } peers { permanent_uuid: "39c96fed49e74c7fa7eb67711b7a131b" member_type: VOTER last_known_addr { host: "127.0.99.61" port: 37451 } } peers { permanent_uuid: "9c8f54d996104f10a8c8e158be67adf4" member_type: VOTER last_known_addr { host: "127.0.99.60" port: 46791 } } }
I20240503 13:59:59.364051 1986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:59.364401 1988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:59.365697 1988 catalog_manager.cc:669] Loaded metadata for table test-workload [id=dfc7bdb5f0c04f5d914597850793cbba]
I20240503 13:59:59.366520 1988 tablet_loader.cc:96] loaded metadata for tablet 4873933fd77643a6bfb683dcaee20c81 (table test-workload [id=dfc7bdb5f0c04f5d914597850793cbba])
I20240503 13:59:59.366737 1988 tablet_loader.cc:96] loaded metadata for tablet 730adc221d8b40e78f9884e21a6f569f (table test-workload [id=dfc7bdb5f0c04f5d914597850793cbba])
I20240503 13:59:59.366901 1988 tablet_loader.cc:96] loaded metadata for tablet 87b96db84666482d984a0fbe1f6d19da (table test-workload [id=dfc7bdb5f0c04f5d914597850793cbba])
I20240503 13:59:59.367010 1988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:59.367450 1988 catalog_manager.cc:1259] Loaded cluster ID: e9cab69efc68448ba90a25b93acd3b93
I20240503 13:59:59.367501 1988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:59.368160 1988 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:59.368593 1988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: Loaded TSK: 0
I20240503 13:59:59.369123 1988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:59.487926 1662 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 13:59:59.488095 1662 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 13:59:59.510074 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 13:59:59.514675 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:59.514902 396 tablet_replica.cc:330] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae: stopping tablet replica
I20240503 13:59:59.515008 396 raft_consensus.cc:2229] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:59.515137 396 raft_consensus.cc:2258] T 87b96db84666482d984a0fbe1f6d19da P 8e2068339c104900a488c7d91231caae [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.528255 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 13:59:59.530490 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 13:59:59.534847 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:59.535061 396 tablet_replica.cc:330] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686: stopping tablet replica
I20240503 13:59:59.535192 396 raft_consensus.cc:2229] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:59.535326 396 raft_consensus.cc:2258] T 4873933fd77643a6bfb683dcaee20c81 P b4d222fec4c3442eab3d2ae568949686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.548128 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 13:59:59.550298 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 13:59:59.554464 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 13:59:59.554663 396 tablet_replica.cc:330] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb: stopping tablet replica
I20240503 13:59:59.554781 396 raft_consensus.cc:2229] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 1 LEADER]: Raft consensus shutting down.
I20240503 13:59:59.554910 396 raft_consensus.cc:2258] T 730adc221d8b40e78f9884e21a6f569f P b1914f7013d7408dbfd70691446382fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.567831 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 13:59:59.569978 396 master.cc:537] Master@127.0.99.61:37451 shutting down...
I20240503 13:59:59.573364 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 13:59:59.573581 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.573647 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 39c96fed49e74c7fa7eb67711b7a131b: stopping tablet replica
I20240503 13:59:59.586942 396 master.cc:552] Master@127.0.99.61:37451 shutdown complete.
I20240503 13:59:59.589650 396 master.cc:537] Master@127.0.99.60:46791 shutting down...
I20240503 13:59:59.593211 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 2 LEADER]: Raft consensus shutting down.
I20240503 13:59:59.593475 396 pending_rounds.cc:62] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: Trying to abort 1 pending ops.
I20240503 13:59:59.593535 396 pending_rounds.cc:68] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: Aborting op as it isn't in flight: id { term: 2 index: 10 } timestamp: 7023594698192429056 op_type: NO_OP noop_request { }
I20240503 13:59:59.593634 396 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240503 13:59:59.593704 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 13:59:59.593796 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9c8f54d996104f10a8c8e158be67adf4: stopping tablet replica
I20240503 13:59:59.596947 396 master.cc:552] Master@127.0.99.60:46791 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4238 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240503 13:59:59.604214 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:40211
I20240503 13:59:59.604585 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:59.607105 1990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:59.607972 1992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:59.608672 1989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:59.609203 396 server_base.cc:1034] running on GCE node
I20240503 13:59:59.609364 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:59.609401 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:59.609421 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744799609420 us; error 0 us; skew 500 ppm
I20240503 13:59:59.610450 396 webserver.cc:460] Webserver started at http://127.0.99.62:46133/ using document root <none> and password file <none>
I20240503 13:59:59.610612 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:59.610666 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:59.610744 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:59.611171 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "22b4f0447bcb4c9f8ceb258824ff588e"
format_stamp: "Formatted at 2024-05-03 13:59:59 on dist-test-slave-6w8v"
I20240503 13:59:59.612816 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 13:59:59.613878 1997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.614112 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 13:59:59.614194 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "22b4f0447bcb4c9f8ceb258824ff588e"
format_stamp: "Formatted at 2024-05-03 13:59:59 on dist-test-slave-6w8v"
I20240503 13:59:59.614271 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:59.628151 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:59.628571 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:59.632905 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:40211
I20240503 13:59:59.635402 2049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:40211 every 8 connection(s)
I20240503 13:59:59.635902 2050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 13:59:59.637820 2050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e: Bootstrap starting.
I20240503 13:59:59.638695 2050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e: Neither blocks nor log segments found. Creating new log.
I20240503 13:59:59.639904 2050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e: No bootstrap required, opened a new log
I20240503 13:59:59.640312 2050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b4f0447bcb4c9f8ceb258824ff588e" member_type: VOTER }
I20240503 13:59:59.640413 2050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 13:59:59.640444 2050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b4f0447bcb4c9f8ceb258824ff588e, State: Initialized, Role: FOLLOWER
I20240503 13:59:59.640573 2050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b4f0447bcb4c9f8ceb258824ff588e" member_type: VOTER }
I20240503 13:59:59.640648 2050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 13:59:59.640687 2050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 13:59:59.640735 2050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 0 FOLLOWER]: Advancing to term 1
I20240503 13:59:59.641472 2050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b4f0447bcb4c9f8ceb258824ff588e" member_type: VOTER }
I20240503 13:59:59.641598 2050 leader_election.cc:302] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22b4f0447bcb4c9f8ceb258824ff588e; no voters:
I20240503 13:59:59.641817 2050 leader_election.cc:288] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 13:59:59.641992 2053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 13:59:59.642443 2053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 1 LEADER]: Becoming Leader. State: Replica: 22b4f0447bcb4c9f8ceb258824ff588e, State: Running, Role: LEADER
I20240503 13:59:59.642679 2053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b4f0447bcb4c9f8ceb258824ff588e" member_type: VOTER }
I20240503 13:59:59.643016 2050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 13:59:59.643147 2055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22b4f0447bcb4c9f8ceb258824ff588e. Latest consensus state: current_term: 1 leader_uuid: "22b4f0447bcb4c9f8ceb258824ff588e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b4f0447bcb4c9f8ceb258824ff588e" member_type: VOTER } }
I20240503 13:59:59.643268 2055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:59.643353 2054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22b4f0447bcb4c9f8ceb258824ff588e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22b4f0447bcb4c9f8ceb258824ff588e" member_type: VOTER } }
I20240503 13:59:59.643486 2054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [sys.catalog]: This master's current role is: LEADER
I20240503 13:59:59.643601 2057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 13:59:59.644615 2057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 13:59:59.644837 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 13:59:59.646462 2057 catalog_manager.cc:1347] Generated new cluster ID: 64f0455585cb46ab9169f1935d758d0c
I20240503 13:59:59.646545 2057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 13:59:59.659543 2057 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 13:59:59.660225 2057 catalog_manager.cc:1504] Loading token signing keys...
I20240503 13:59:59.666683 2057 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e: Generated new TSK 0
I20240503 13:59:59.666901 2057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 13:59:59.677456 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:59.680567 2071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:59.680964 2072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:59.681911 2074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:59.682652 396 server_base.cc:1034] running on GCE node
I20240503 13:59:59.682829 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:59.682866 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:59.682881 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744799682881 us; error 0 us; skew 500 ppm
I20240503 13:59:59.684048 396 webserver.cc:460] Webserver started at http://127.0.99.1:45225/ using document root <none> and password file <none>
I20240503 13:59:59.684224 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:59.684276 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:59.684336 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:59.684736 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-0-root/instance:
uuid: "731f0964cb534e7c881a2d7841da7ba3"
format_stamp: "Formatted at 2024-05-03 13:59:59 on dist-test-slave-6w8v"
I20240503 13:59:59.686312 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:59.688156 2079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.688534 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:59.688649 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-0-root
uuid: "731f0964cb534e7c881a2d7841da7ba3"
format_stamp: "Formatted at 2024-05-03 13:59:59 on dist-test-slave-6w8v"
I20240503 13:59:59.688742 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:59.697918 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:59.698343 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:59.698673 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:59.699256 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:59.699299 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.699333 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:59.699348 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.704018 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:43071
I20240503 13:59:59.704049 2142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:43071 every 8 connection(s)
I20240503 13:59:59.704907 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:59.707870 2147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:59.708326 2148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:59.709213 2150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:59.709892 2143 heartbeater.cc:347] Connected to a master server at 127.0.99.62:40211
I20240503 13:59:59.710031 2143 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:59.710055 396 server_base.cc:1034] running on GCE node
I20240503 13:59:59.710247 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:59.710285 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:59.710306 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744799710305 us; error 0 us; skew 500 ppm
I20240503 13:59:59.710356 2143 heartbeater.cc:511] Master 127.0.99.62:40211 requested a full tablet report, sending...
I20240503 13:59:59.711211 2014 ts_manager.cc:194] Registered new tserver with Master: 731f0964cb534e7c881a2d7841da7ba3 (127.0.99.1:43071)
I20240503 13:59:59.711630 396 webserver.cc:460] Webserver started at http://127.0.99.2:38991/ using document root <none> and password file <none>
I20240503 13:59:59.711805 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:59.711865 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:59.711941 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:59.712342 2014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55112
I20240503 13:59:59.712455 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-1-root/instance:
uuid: "cec3afdbd1714558b18ca860d93dd7a5"
format_stamp: "Formatted at 2024-05-03 13:59:59 on dist-test-slave-6w8v"
I20240503 13:59:59.714185 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:59.715515 2155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.715823 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:59.715906 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-1-root
uuid: "cec3afdbd1714558b18ca860d93dd7a5"
format_stamp: "Formatted at 2024-05-03 13:59:59 on dist-test-slave-6w8v"
I20240503 13:59:59.715991 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:59.736328 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:59.736800 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:59.737121 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:59.737639 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:59.737680 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.737727 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:59.737761 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.742465 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:39041
I20240503 13:59:59.742592 2218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:39041 every 8 connection(s)
I20240503 13:59:59.743602 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 13:59:59.746587 2223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 13:59:59.747494 2222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:59.748164 2219 heartbeater.cc:347] Connected to a master server at 127.0.99.62:40211
I20240503 13:59:59.748323 2219 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:59.748368 396 server_base.cc:1034] running on GCE node
I20240503 13:59:59.748656 2219 heartbeater.cc:511] Master 127.0.99.62:40211 requested a full tablet report, sending...
W20240503 13:59:59.748903 2225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 13:59:59.749362 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 13:59:59.749415 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 13:59:59.749436 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744799749436 us; error 0 us; skew 500 ppm
I20240503 13:59:59.749471 2014 ts_manager.cc:194] Registered new tserver with Master: cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2:39041)
I20240503 13:59:59.750097 2014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55114
I20240503 13:59:59.750588 396 webserver.cc:460] Webserver started at http://127.0.99.3:40379/ using document root <none> and password file <none>
I20240503 13:59:59.750766 396 fs_manager.cc:359] Metadata directory not provided
I20240503 13:59:59.750823 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 13:59:59.750905 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 13:59:59.751310 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-2-root/instance:
uuid: "761a94997dd24b8d8b0edf66b4d66088"
format_stamp: "Formatted at 2024-05-03 13:59:59 on dist-test-slave-6w8v"
I20240503 13:59:59.752971 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 13:59:59.754089 2230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.754457 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 13:59:59.754542 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-2-root
uuid: "761a94997dd24b8d8b0edf66b4d66088"
format_stamp: "Formatted at 2024-05-03 13:59:59 on dist-test-slave-6w8v"
I20240503 13:59:59.754628 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 13:59:59.768779 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 13:59:59.769222 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 13:59:59.769516 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 13:59:59.770009 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 13:59:59.770049 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.770094 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 13:59:59.770121 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 13:59:59.774490 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:33479
I20240503 13:59:59.774535 2293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:33479 every 8 connection(s)
I20240503 13:59:59.779220 2294 heartbeater.cc:347] Connected to a master server at 127.0.99.62:40211
I20240503 13:59:59.779346 2294 heartbeater.cc:464] Registering TS with master...
I20240503 13:59:59.779633 2294 heartbeater.cc:511] Master 127.0.99.62:40211 requested a full tablet report, sending...
I20240503 13:59:59.780370 2014 ts_manager.cc:194] Registered new tserver with Master: 761a94997dd24b8d8b0edf66b4d66088 (127.0.99.3:33479)
I20240503 13:59:59.780596 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005695391s
I20240503 13:59:59.781109 2014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55130
I20240503 14:00:00.713466 2143 heartbeater.cc:503] Master 127.0.99.62:40211 was elected leader, sending a full tablet report...
I20240503 14:00:00.751217 2219 heartbeater.cc:503] Master 127.0.99.62:40211 was elected leader, sending a full tablet report...
I20240503 14:00:00.782155 2294 heartbeater.cc:503] Master 127.0.99.62:40211 was elected leader, sending a full tablet report...
I20240503 14:00:00.807476 396 test_util.cc:268] Using random seed: 582715953
I20240503 14:00:00.820755 2014 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55142:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:00:00.821271 2014 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:00:00.831882 2257 tablet_service.cc:1467] Processing CreateTablet for tablet a43205fc9c9d44c5aae457fb61184835 (DEFAULT_TABLE table=test-workload [id=1e6614c5f0e1428193744fafefb51c72]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:00:00.832051 2183 tablet_service.cc:1467] Processing CreateTablet for tablet 73cc3fa988034bb08f21b1613ad25b2e (DEFAULT_TABLE table=test-workload [id=1e6614c5f0e1428193744fafefb51c72]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:00:00.832306 2257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43205fc9c9d44c5aae457fb61184835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.832351 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cc3fa988034bb08f21b1613ad25b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.832990 2182 tablet_service.cc:1467] Processing CreateTablet for tablet a43205fc9c9d44c5aae457fb61184835 (DEFAULT_TABLE table=test-workload [id=1e6614c5f0e1428193744fafefb51c72]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:00:00.833246 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43205fc9c9d44c5aae457fb61184835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.834517 2314 tablet_bootstrap.cc:492] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088: Bootstrap starting.
I20240503 14:00:00.834923 2107 tablet_service.cc:1467] Processing CreateTablet for tablet 73cc3fa988034bb08f21b1613ad25b2e (DEFAULT_TABLE table=test-workload [id=1e6614c5f0e1428193744fafefb51c72]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:00:00.835072 2315 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: Bootstrap starting.
I20240503 14:00:00.835166 2107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cc3fa988034bb08f21b1613ad25b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.835532 2314 tablet_bootstrap.cc:654] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:00.835600 2106 tablet_service.cc:1467] Processing CreateTablet for tablet a43205fc9c9d44c5aae457fb61184835 (DEFAULT_TABLE table=test-workload [id=1e6614c5f0e1428193744fafefb51c72]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240503 14:00:00.835841 2106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a43205fc9c9d44c5aae457fb61184835. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.836205 2315 tablet_bootstrap.cc:654] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:00.837167 2318 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3: Bootstrap starting.
I20240503 14:00:00.837503 2315 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: No bootstrap required, opened a new log
I20240503 14:00:00.837585 2315 ts_tablet_manager.cc:1367] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:00.837795 2314 tablet_bootstrap.cc:492] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088: No bootstrap required, opened a new log
I20240503 14:00:00.837867 2314 ts_tablet_manager.cc:1367] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:00:00.838173 2315 raft_consensus.cc:348] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.838265 2314 raft_consensus.cc:348] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.838327 2315 raft_consensus.cc:374] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:00.831882 2258 tablet_service.cc:1467] Processing CreateTablet for tablet 73cc3fa988034bb08f21b1613ad25b2e (DEFAULT_TABLE table=test-workload [id=1e6614c5f0e1428193744fafefb51c72]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240503 14:00:00.838363 2314 raft_consensus.cc:374] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:00.838416 2314 raft_consensus.cc:729] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761a94997dd24b8d8b0edf66b4d66088, State: Initialized, Role: FOLLOWER
I20240503 14:00:00.838367 2315 raft_consensus.cc:729] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec3afdbd1714558b18ca860d93dd7a5, State: Initialized, Role: FOLLOWER
I20240503 14:00:00.838573 2258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cc3fa988034bb08f21b1613ad25b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:00.838541 2314 consensus_queue.cc:260] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.838593 2315 consensus_queue.cc:260] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.838898 2314 ts_tablet_manager.cc:1398] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:00.838948 2315 ts_tablet_manager.cc:1398] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:00.838075 2318 tablet_bootstrap.cc:654] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:00.839447 2315 tablet_bootstrap.cc:492] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5: Bootstrap starting.
I20240503 14:00:00.840373 2315 tablet_bootstrap.cc:654] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:00.840705 2314 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088: Bootstrap starting.
I20240503 14:00:00.841445 2315 tablet_bootstrap.cc:492] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5: No bootstrap required, opened a new log
I20240503 14:00:00.841531 2315 ts_tablet_manager.cc:1367] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:00.841693 2314 tablet_bootstrap.cc:654] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:00.841971 2315 raft_consensus.cc:348] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.842334 2315 raft_consensus.cc:374] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:00.842381 2315 raft_consensus.cc:729] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cec3afdbd1714558b18ca860d93dd7a5, State: Initialized, Role: FOLLOWER
I20240503 14:00:00.842501 2315 consensus_queue.cc:260] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.842671 2318 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3: No bootstrap required, opened a new log
I20240503 14:00:00.842813 2318 ts_tablet_manager.cc:1367] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:00:00.842886 2315 ts_tablet_manager.cc:1398] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:00.843425 2318 raft_consensus.cc:348] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.843523 2318 raft_consensus.cc:374] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:00.843554 2318 raft_consensus.cc:729] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731f0964cb534e7c881a2d7841da7ba3, State: Initialized, Role: FOLLOWER
I20240503 14:00:00.843829 2318 consensus_queue.cc:260] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.844295 2318 ts_tablet_manager.cc:1398] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240503 14:00:00.844568 2318 tablet_bootstrap.cc:492] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3: Bootstrap starting.
I20240503 14:00:00.844771 2314 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088: No bootstrap required, opened a new log
I20240503 14:00:00.844889 2314 ts_tablet_manager.cc:1367] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:00:00.845337 2314 raft_consensus.cc:348] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.845445 2314 raft_consensus.cc:374] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:00.845504 2314 raft_consensus.cc:729] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 761a94997dd24b8d8b0edf66b4d66088, State: Initialized, Role: FOLLOWER
I20240503 14:00:00.845702 2318 tablet_bootstrap.cc:654] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:00.845655 2314 consensus_queue.cc:260] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.846004 2314 ts_tablet_manager.cc:1398] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:00.847193 2318 tablet_bootstrap.cc:492] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3: No bootstrap required, opened a new log
I20240503 14:00:00.847323 2318 ts_tablet_manager.cc:1367] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:00:00.847775 2318 raft_consensus.cc:348] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.847874 2318 raft_consensus.cc:374] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:00.847918 2318 raft_consensus.cc:729] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731f0964cb534e7c881a2d7841da7ba3, State: Initialized, Role: FOLLOWER
I20240503 14:00:00.848037 2318 consensus_queue.cc:260] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.848273 2318 ts_tablet_manager.cc:1398] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:00.892266 2319 raft_consensus.cc:482] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:00.892418 2319 raft_consensus.cc:504] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.892925 2319 leader_election.cc:288] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2:39041), 731f0964cb534e7c881a2d7841da7ba3 (127.0.99.1:43071)
I20240503 14:00:00.893251 2324 raft_consensus.cc:482] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:00.893429 2324 raft_consensus.cc:504] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.894860 2324 leader_election.cc:288] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2:39041), 761a94997dd24b8d8b0edf66b4d66088 (127.0.99.3:33479)
I20240503 14:00:00.897650 2193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cc3fa988034bb08f21b1613ad25b2e" candidate_uuid: "761a94997dd24b8d8b0edf66b4d66088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec3afdbd1714558b18ca860d93dd7a5" is_pre_election: true
I20240503 14:00:00.897650 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cc3fa988034bb08f21b1613ad25b2e" candidate_uuid: "761a94997dd24b8d8b0edf66b4d66088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731f0964cb534e7c881a2d7841da7ba3" is_pre_election: true
I20240503 14:00:00.897835 2117 raft_consensus.cc:2454] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761a94997dd24b8d8b0edf66b4d66088 in term 0.
I20240503 14:00:00.897836 2193 raft_consensus.cc:2454] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 761a94997dd24b8d8b0edf66b4d66088 in term 0.
I20240503 14:00:00.898002 2192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cc3fa988034bb08f21b1613ad25b2e" candidate_uuid: "731f0964cb534e7c881a2d7841da7ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec3afdbd1714558b18ca860d93dd7a5" is_pre_election: true
I20240503 14:00:00.898088 2192 raft_consensus.cc:2454] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 731f0964cb534e7c881a2d7841da7ba3 in term 0.
I20240503 14:00:00.898231 2231 leader_election.cc:302] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761a94997dd24b8d8b0edf66b4d66088, cec3afdbd1714558b18ca860d93dd7a5; no voters:
I20240503 14:00:00.898330 2080 leader_election.cc:302] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731f0964cb534e7c881a2d7841da7ba3, cec3afdbd1714558b18ca860d93dd7a5; no voters:
I20240503 14:00:00.898386 2319 raft_consensus.cc:2781] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:00.898442 2319 raft_consensus.cc:482] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:00.898458 2324 raft_consensus.cc:2781] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:00.898478 2319 raft_consensus.cc:3037] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:00.898501 2324 raft_consensus.cc:482] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:00.898535 2324 raft_consensus.cc:3037] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:00.898702 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cc3fa988034bb08f21b1613ad25b2e" candidate_uuid: "731f0964cb534e7c881a2d7841da7ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761a94997dd24b8d8b0edf66b4d66088" is_pre_election: true
I20240503 14:00:00.899441 2319 raft_consensus.cc:504] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.899443 2324 raft_consensus.cc:504] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.899679 2268 raft_consensus.cc:2379] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 731f0964cb534e7c881a2d7841da7ba3 in current term 1: Already voted for candidate 761a94997dd24b8d8b0edf66b4d66088 in this term.
I20240503 14:00:00.899809 2319 leader_election.cc:288] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [CANDIDATE]: Term 1 election: Requested vote from peers cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2:39041), 731f0964cb534e7c881a2d7841da7ba3 (127.0.99.1:43071)
I20240503 14:00:00.900038 2192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cc3fa988034bb08f21b1613ad25b2e" candidate_uuid: "731f0964cb534e7c881a2d7841da7ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec3afdbd1714558b18ca860d93dd7a5"
I20240503 14:00:00.900138 2192 raft_consensus.cc:3037] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:00.900166 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cc3fa988034bb08f21b1613ad25b2e" candidate_uuid: "761a94997dd24b8d8b0edf66b4d66088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731f0964cb534e7c881a2d7841da7ba3"
I20240503 14:00:00.900167 2193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cc3fa988034bb08f21b1613ad25b2e" candidate_uuid: "761a94997dd24b8d8b0edf66b4d66088" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cec3afdbd1714558b18ca860d93dd7a5"
I20240503 14:00:00.900321 2117 raft_consensus.cc:2379] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 761a94997dd24b8d8b0edf66b4d66088 in current term 1: Already voted for candidate 731f0964cb534e7c881a2d7841da7ba3 in this term.
I20240503 14:00:00.900549 2231 leader_election.cc:302] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 761a94997dd24b8d8b0edf66b4d66088; no voters: 731f0964cb534e7c881a2d7841da7ba3, cec3afdbd1714558b18ca860d93dd7a5
I20240503 14:00:00.900724 2319 raft_consensus.cc:2726] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:00:00.901032 2192 raft_consensus.cc:2454] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 731f0964cb534e7c881a2d7841da7ba3 in term 1.
I20240503 14:00:00.901242 2324 leader_election.cc:288] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [CANDIDATE]: Term 1 election: Requested vote from peers cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2:39041), 761a94997dd24b8d8b0edf66b4d66088 (127.0.99.3:33479)
I20240503 14:00:00.901269 2080 leader_election.cc:302] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731f0964cb534e7c881a2d7841da7ba3, cec3afdbd1714558b18ca860d93dd7a5; no voters:
I20240503 14:00:00.901405 2324 raft_consensus.cc:2781] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:00.901512 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cc3fa988034bb08f21b1613ad25b2e" candidate_uuid: "731f0964cb534e7c881a2d7841da7ba3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761a94997dd24b8d8b0edf66b4d66088"
I20240503 14:00:00.901607 2324 raft_consensus.cc:686] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 LEADER]: Becoming Leader. State: Replica: 731f0964cb534e7c881a2d7841da7ba3, State: Running, Role: LEADER
I20240503 14:00:00.901630 2268 raft_consensus.cc:2379] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 731f0964cb534e7c881a2d7841da7ba3 in current term 1: Already voted for candidate 761a94997dd24b8d8b0edf66b4d66088 in this term.
I20240503 14:00:00.901777 2324 consensus_queue.cc:237] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } }
I20240503 14:00:00.903265 2013 catalog_manager.cc:5484] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 731f0964cb534e7c881a2d7841da7ba3 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "731f0964cb534e7c881a2d7841da7ba3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } health_report { overall_health: HEALTHY } } }
I20240503 14:00:00.933167 2320 raft_consensus.cc:482] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:00.933347 2320 raft_consensus.cc:504] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.933907 2320 leader_election.cc:288] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 731f0964cb534e7c881a2d7841da7ba3 (127.0.99.1:43071), 761a94997dd24b8d8b0edf66b4d66088 (127.0.99.3:33479)
I20240503 14:00:00.937393 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43205fc9c9d44c5aae457fb61184835" candidate_uuid: "cec3afdbd1714558b18ca860d93dd7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761a94997dd24b8d8b0edf66b4d66088" is_pre_election: true
I20240503 14:00:00.937435 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43205fc9c9d44c5aae457fb61184835" candidate_uuid: "cec3afdbd1714558b18ca860d93dd7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731f0964cb534e7c881a2d7841da7ba3" is_pre_election: true
I20240503 14:00:00.937569 2268 raft_consensus.cc:2454] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec3afdbd1714558b18ca860d93dd7a5 in term 0.
I20240503 14:00:00.937569 2117 raft_consensus.cc:2454] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cec3afdbd1714558b18ca860d93dd7a5 in term 0.
I20240503 14:00:00.937863 2159 leader_election.cc:302] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 761a94997dd24b8d8b0edf66b4d66088, cec3afdbd1714558b18ca860d93dd7a5; no voters:
I20240503 14:00:00.938040 2320 raft_consensus.cc:2781] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:00.938100 2320 raft_consensus.cc:482] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:00.938143 2320 raft_consensus.cc:3037] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:00.938966 2320 raft_consensus.cc:504] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.939314 2320 leader_election.cc:288] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [CANDIDATE]: Term 1 election: Requested vote from peers 731f0964cb534e7c881a2d7841da7ba3 (127.0.99.1:43071), 761a94997dd24b8d8b0edf66b4d66088 (127.0.99.3:33479)
I20240503 14:00:00.939687 2268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43205fc9c9d44c5aae457fb61184835" candidate_uuid: "cec3afdbd1714558b18ca860d93dd7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "761a94997dd24b8d8b0edf66b4d66088"
I20240503 14:00:00.939710 2117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a43205fc9c9d44c5aae457fb61184835" candidate_uuid: "cec3afdbd1714558b18ca860d93dd7a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "731f0964cb534e7c881a2d7841da7ba3"
I20240503 14:00:00.939782 2268 raft_consensus.cc:3037] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:00.939814 2117 raft_consensus.cc:3037] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:00.940698 2268 raft_consensus.cc:2454] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec3afdbd1714558b18ca860d93dd7a5 in term 1.
I20240503 14:00:00.940701 2117 raft_consensus.cc:2454] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cec3afdbd1714558b18ca860d93dd7a5 in term 1.
I20240503 14:00:00.941006 2156 leader_election.cc:302] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 731f0964cb534e7c881a2d7841da7ba3, cec3afdbd1714558b18ca860d93dd7a5; no voters:
I20240503 14:00:00.941162 2320 raft_consensus.cc:2781] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:00.941383 2320 raft_consensus.cc:686] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 1 LEADER]: Becoming Leader. State: Replica: cec3afdbd1714558b18ca860d93dd7a5, State: Running, Role: LEADER
I20240503 14:00:00.941519 2320 consensus_queue.cc:237] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } }
I20240503 14:00:00.943012 2013 catalog_manager.cc:5484] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "cec3afdbd1714558b18ca860d93dd7a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:01.336647 2320 consensus_queue.cc:1035] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.340082 2340 consensus_queue.cc:1035] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.377406 2324 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.380694 2345 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:01.644930 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:01.645375 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:01.645443 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:03.645740 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:03.646147 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:03.646216 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:05.646548 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:05.646944 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:05.647011 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:07.647374 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:07.647892 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:07.647966 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:09.648317 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:09.648727 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:09.648795 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:11.649087 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:11.649513 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:11.649580 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:12.005631 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:00:12.008821 396 server_base.cc:1034] running on GCE node
W20240503 14:00:12.009269 2353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:12.010125 2355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:12.010833 2352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:12.011086 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:12.011134 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:12.011150 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744812011150 us; error 0 us; skew 500 ppm
I20240503 14:00:12.012214 396 webserver.cc:460] Webserver started at http://127.0.99.4:46375/ using document root <none> and password file <none>
I20240503 14:00:12.012353 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:12.012400 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:12.012465 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:12.012861 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-3-root/instance:
uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2"
format_stamp: "Formatted at 2024-05-03 14:00:12 on dist-test-slave-6w8v"
I20240503 14:00:12.014283 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:12.015246 2360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:12.015492 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:12.015576 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-3-root
uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2"
format_stamp: "Formatted at 2024-05-03 14:00:12 on dist-test-slave-6w8v"
I20240503 14:00:12.015661 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:12.028331 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:12.028707 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:12.028995 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:12.029600 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:12.029649 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:12.029701 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:12.029729 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:12.034271 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:34047
I20240503 14:00:12.034300 2423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:34047 every 8 connection(s)
I20240503 14:00:12.039136 2424 heartbeater.cc:347] Connected to a master server at 127.0.99.62:40211
I20240503 14:00:12.039273 2424 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:12.039531 2424 heartbeater.cc:511] Master 127.0.99.62:40211 requested a full tablet report, sending...
I20240503 14:00:12.040242 2014 ts_manager.cc:194] Registered new tserver with Master: 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4:34047)
I20240503 14:00:12.040803 2014 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42156
I20240503 14:00:12.661103 2117 consensus_queue.cc:237] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: true } }
I20240503 14:00:12.662786 2192 raft_consensus.cc:1261] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Refusing update from remote peer 731f0964cb534e7c881a2d7841da7ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.662957 2268 raft_consensus.cc:1261] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Refusing update from remote peer 731f0964cb534e7c881a2d7841da7ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:00:12.663220 2432 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:12.663481 2431 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:00:12.665318 2431 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: true } } }
I20240503 14:00:12.665597 2268 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: true } } }
I20240503 14:00:12.665680 2192 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: true } } }
I20240503 14:00:12.667883 2014 catalog_manager.cc:5484] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 reported cstate change: config changed from index -1 to 2, NON_VOTER 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "731f0964cb534e7c881a2d7841da7ba3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: true } } }
W20240503 14:00:12.668723 2083 consensus_peers.cc:485] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 -> Peer 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4:34047): Couldn't send request to peer 6e8267c4f7a94ebba2d8121ce3dbf9e2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73cc3fa988034bb08f21b1613ad25b2e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:00:13.041779 2424 heartbeater.cc:503] Master 127.0.99.62:40211 was elected leader, sending a full tablet report...
I20240503 14:00:13.256690 2442 ts_tablet_manager.cc:902] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: Initiating tablet copy from peer 731f0964cb534e7c881a2d7841da7ba3 (127.0.99.1:43071)
I20240503 14:00:13.257592 2442 tablet_copy_client.cc:308] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:43071
I20240503 14:00:13.261881 2127 tablet_copy_service.cc:140] P 731f0964cb534e7c881a2d7841da7ba3: Received BeginTabletCopySession request for tablet 73cc3fa988034bb08f21b1613ad25b2e from peer 6e8267c4f7a94ebba2d8121ce3dbf9e2 ({username='slave'} at 127.0.0.1:51230)
I20240503 14:00:13.262084 2127 tablet_copy_service.cc:161] P 731f0964cb534e7c881a2d7841da7ba3: Beginning new tablet copy session on tablet 73cc3fa988034bb08f21b1613ad25b2e from peer 6e8267c4f7a94ebba2d8121ce3dbf9e2 at {username='slave'} at 127.0.0.1:51230: session id = 6e8267c4f7a94ebba2d8121ce3dbf9e2-73cc3fa988034bb08f21b1613ad25b2e
I20240503 14:00:13.263665 2127 tablet_copy_source_session.cc:205] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:00:13.265252 2442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cc3fa988034bb08f21b1613ad25b2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:13.267809 2442 tablet_copy_client.cc:783] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: tablet copy: Starting download of 0 data blocks...
I20240503 14:00:13.268003 2442 tablet_copy_client.cc:647] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: tablet copy: Starting download of 1 WAL segments...
I20240503 14:00:13.269709 2442 tablet_copy_client.cc:523] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:00:13.271533 2442 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: Bootstrap starting.
I20240503 14:00:13.276244 2442 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:00:13.276564 2442 tablet_bootstrap.cc:492] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: Bootstrap complete.
I20240503 14:00:13.276753 2442 ts_tablet_manager.cc:1367] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.006s
I20240503 14:00:13.277205 2442 raft_consensus.cc:348] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: true } }
I20240503 14:00:13.277333 2442 raft_consensus.cc:729] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6e8267c4f7a94ebba2d8121ce3dbf9e2, State: Initialized, Role: LEARNER
I20240503 14:00:13.277438 2442 consensus_queue.cc:260] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: true } }
I20240503 14:00:13.277843 2442 ts_tablet_manager.cc:1398] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240503 14:00:13.278373 2127 tablet_copy_service.cc:342] P 731f0964cb534e7c881a2d7841da7ba3: Request end of tablet copy session 6e8267c4f7a94ebba2d8121ce3dbf9e2-73cc3fa988034bb08f21b1613ad25b2e received from {username='slave'} at 127.0.0.1:51230
I20240503 14:00:13.278476 2127 tablet_copy_service.cc:433] P 731f0964cb534e7c881a2d7841da7ba3: ending tablet copy session 6e8267c4f7a94ebba2d8121ce3dbf9e2-73cc3fa988034bb08f21b1613ad25b2e on tablet 73cc3fa988034bb08f21b1613ad25b2e with peer 6e8267c4f7a94ebba2d8121ce3dbf9e2
I20240503 14:00:13.649814 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:13.650202 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:13.650275 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:13.653827 2398 raft_consensus.cc:1203] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:00:14.185178 2452 raft_consensus.cc:1050] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3: attempting to promote NON_VOTER 6e8267c4f7a94ebba2d8121ce3dbf9e2 to VOTER
I20240503 14:00:14.185629 2452 consensus_queue.cc:237] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } }
I20240503 14:00:14.187011 2398 raft_consensus.cc:1261] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 LEARNER]: Refusing update from remote peer 731f0964cb534e7c881a2d7841da7ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:14.187201 2192 raft_consensus.cc:1261] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Refusing update from remote peer 731f0964cb534e7c881a2d7841da7ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:14.187424 2456 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:14.187938 2455 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:14.187948 2268 raft_consensus.cc:1261] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Refusing update from remote peer 731f0964cb534e7c881a2d7841da7ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:00:14.188416 2456 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:00:14.190156 2455 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } } }
I20240503 14:00:14.190547 2268 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } } }
I20240503 14:00:14.191814 2192 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } } }
I20240503 14:00:14.192875 2013 catalog_manager.cc:5484] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 reported cstate change: config changed from index 2 to 3, 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "731f0964cb534e7c881a2d7841da7ba3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } } }
I20240503 14:00:14.193627 2398 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6e8267c4f7a94ebba2d8121ce3dbf9e2 (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cec3afdbd1714558b18ca860d93dd7a5" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 39041 } attrs { replace: true } } peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } } }
I20240503 14:00:14.196689 2117 consensus_queue.cc:237] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } }
I20240503 14:00:14.197751 2398 raft_consensus.cc:1261] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 FOLLOWER]: Refusing update from remote peer 731f0964cb534e7c881a2d7841da7ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:14.197751 2268 raft_consensus.cc:1261] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Refusing update from remote peer 731f0964cb534e7c881a2d7841da7ba3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:00:14.198200 2455 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:14.198688 2455 consensus_queue.cc:1035] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:00:14.199916 2453 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } } }
I20240503 14:00:14.201371 1998 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73cc3fa988034bb08f21b1613ad25b2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:00:14.201483 2268 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } } }
I20240503 14:00:14.201813 2398 raft_consensus.cc:2932] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } } }
I20240503 14:00:14.202008 2012 catalog_manager.cc:5484] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 reported cstate change: config changed from index 3 to 4, VOTER cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "731f0964cb534e7c881a2d7841da7ba3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "761a94997dd24b8d8b0edf66b4d66088" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "731f0964cb534e7c881a2d7841da7ba3" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 43071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e8267c4f7a94ebba2d8121ce3dbf9e2" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 34047 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:14.204890 2182 tablet_service.cc:1514] Processing DeleteTablet for tablet 73cc3fa988034bb08f21b1613ad25b2e with delete_type TABLET_DATA_TOMBSTONED (TS cec3afdbd1714558b18ca860d93dd7a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:35564
I20240503 14:00:14.205431 2466 tablet_replica.cc:330] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: stopping tablet replica
I20240503 14:00:14.205574 2466 raft_consensus.cc:2229] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:14.205744 2466 raft_consensus.cc:2258] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:14.206341 2466 ts_tablet_manager.cc:1875] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:00:14.208343 2466 ts_tablet_manager.cc:1888] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:00:14.208446 2466 log.cc:1196] T 73cc3fa988034bb08f21b1613ad25b2e P cec3afdbd1714558b18ca860d93dd7a5: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714744785267298-396-0/minicluster-data/ts-1-root/wals/73cc3fa988034bb08f21b1613ad25b2e
I20240503 14:00:14.209010 1998 catalog_manager.cc:4830] TS cec3afdbd1714558b18ca860d93dd7a5 (127.0.99.2:39041): tablet 73cc3fa988034bb08f21b1613ad25b2e (table test-workload [id=1e6614c5f0e1428193744fafefb51c72]) successfully deleted
I20240503 14:00:15.650671 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:15.651067 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:15.651129 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:17.651501 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:17.651917 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:17.651978 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:19.652369 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:19.652747 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:19.652806 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:21.653170 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:21.653570 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:21.653630 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:23.653990 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:23.654449 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:23.654520 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:25.654939 2068 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:25.655391 2068 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:25.655459 2068 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:26.369336 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 14:00:26.372833 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:00:26.373044 396 tablet_replica.cc:330] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3: stopping tablet replica
I20240503 14:00:26.373180 396 raft_consensus.cc:2229] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.373310 396 raft_consensus.cc:2258] T a43205fc9c9d44c5aae457fb61184835 P 731f0964cb534e7c881a2d7841da7ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.373785 396 tablet_replica.cc:330] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3: stopping tablet replica
I20240503 14:00:26.373862 396 raft_consensus.cc:2229] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:00:26.374047 396 raft_consensus.cc:2258] T 73cc3fa988034bb08f21b1613ad25b2e P 731f0964cb534e7c881a2d7841da7ba3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.387203 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 14:00:26.389510 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 14:00:26.393158 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:00:26.393380 396 tablet_replica.cc:330] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5: stopping tablet replica
I20240503 14:00:26.393496 396 raft_consensus.cc:2229] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:00:26.393702 396 raft_consensus.cc:2258] T a43205fc9c9d44c5aae457fb61184835 P cec3afdbd1714558b18ca860d93dd7a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.406917 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 14:00:26.409211 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 14:00:26.412984 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:00:26.413218 396 tablet_replica.cc:330] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088: stopping tablet replica
I20240503 14:00:26.413337 396 raft_consensus.cc:2229] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.413499 396 raft_consensus.cc:2258] T 73cc3fa988034bb08f21b1613ad25b2e P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.413977 396 tablet_replica.cc:330] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088: stopping tablet replica
I20240503 14:00:26.414054 396 raft_consensus.cc:2229] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.414173 396 raft_consensus.cc:2258] T a43205fc9c9d44c5aae457fb61184835 P 761a94997dd24b8d8b0edf66b4d66088 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.426820 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 14:00:26.428989 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 14:00:26.432160 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:00:26.432335 396 tablet_replica.cc:330] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2: stopping tablet replica
I20240503 14:00:26.432443 396 raft_consensus.cc:2229] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:00:26.432581 396 raft_consensus.cc:2258] T 73cc3fa988034bb08f21b1613ad25b2e P 6e8267c4f7a94ebba2d8121ce3dbf9e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.445026 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 14:00:26.447185 396 master.cc:537] Master@127.0.99.62:40211 shutting down...
I20240503 14:00:26.450398 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:00:26.450599 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:26.450663 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22b4f0447bcb4c9f8ceb258824ff588e: stopping tablet replica
I20240503 14:00:26.463214 396 master.cc:552] Master@127.0.99.62:40211 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26866 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240503 14:00:26.470412 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:45725
I20240503 14:00:26.470769 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:26.473940 2470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:26.474368 2472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:26.475178 396 server_base.cc:1034] running on GCE node
W20240503 14:00:26.475200 2469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:26.475548 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:26.475626 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:26.475643 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744826475643 us; error 0 us; skew 500 ppm
I20240503 14:00:26.481611 396 webserver.cc:460] Webserver started at http://127.0.99.62:36467/ using document root <none> and password file <none>
I20240503 14:00:26.481779 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:26.481829 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:26.481892 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:26.482297 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8"
format_stamp: "Formatted at 2024-05-03 14:00:26 on dist-test-slave-6w8v"
I20240503 14:00:26.483896 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 14:00:26.484966 2477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:26.485251 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:26.485323 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8"
format_stamp: "Formatted at 2024-05-03 14:00:26 on dist-test-slave-6w8v"
I20240503 14:00:26.485401 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:26.500519 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:26.500919 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:26.505565 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:45725
I20240503 14:00:26.508221 2529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:45725 every 8 connection(s)
I20240503 14:00:26.508634 2530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:26.510550 2530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8: Bootstrap starting.
I20240503 14:00:26.511363 2530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:26.512629 2530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8: No bootstrap required, opened a new log
I20240503 14:00:26.513032 2530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8" member_type: VOTER }
I20240503 14:00:26.513130 2530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:26.513178 2530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47a76a9b4b7c4cdfbd0b4ff00d97dae8, State: Initialized, Role: FOLLOWER
I20240503 14:00:26.513317 2530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8" member_type: VOTER }
I20240503 14:00:26.513399 2530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:00:26.513429 2530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:00:26.513463 2530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:26.514197 2530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8" member_type: VOTER }
I20240503 14:00:26.514326 2530 leader_election.cc:302] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47a76a9b4b7c4cdfbd0b4ff00d97dae8; no voters:
I20240503 14:00:26.514497 2530 leader_election.cc:288] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:00:26.514636 2533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:26.514853 2533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 1 LEADER]: Becoming Leader. State: Replica: 47a76a9b4b7c4cdfbd0b4ff00d97dae8, State: Running, Role: LEADER
I20240503 14:00:26.514945 2530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:00:26.514990 2533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8" member_type: VOTER }
I20240503 14:00:26.515471 2535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47a76a9b4b7c4cdfbd0b4ff00d97dae8. Latest consensus state: current_term: 1 leader_uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8" member_type: VOTER } }
I20240503 14:00:26.515601 2535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:26.515442 2534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47a76a9b4b7c4cdfbd0b4ff00d97dae8" member_type: VOTER } }
I20240503 14:00:26.515691 2534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:26.515969 2540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:00:26.516920 2540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:00:26.517141 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:00:26.519023 2540 catalog_manager.cc:1347] Generated new cluster ID: e3a6efe735b843f8b20ad5e91d07ba45
I20240503 14:00:26.519104 2540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:00:26.528949 2540 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:00:26.529817 2540 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:00:26.540171 2540 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8: Generated new TSK 0
I20240503 14:00:26.540365 2540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:00:26.549182 396 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 8.4663e-05s
I20240503 14:00:38.628937 396 master.cc:537] Master@127.0.99.62:45725 shutting down...
I20240503 14:00:38.631903 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:00:38.632117 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:38.632180 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 47a76a9b4b7c4cdfbd0b4ff00d97dae8: stopping tablet replica
I20240503 14:00:38.644418 396 master.cc:552] Master@127.0.99.62:45725 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12178 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240503 14:00:38.648756 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:43825
I20240503 14:00:38.649107 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:38.651813 2555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:38.652335 2553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:38.652892 2552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:38.653743 396 server_base.cc:1034] running on GCE node
I20240503 14:00:38.653929 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:38.653965 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:38.653986 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744838653985 us; error 0 us; skew 500 ppm
I20240503 14:00:38.655032 396 webserver.cc:460] Webserver started at http://127.0.99.62:37611/ using document root <none> and password file <none>
I20240503 14:00:38.655191 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:38.655246 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:38.655326 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:38.655783 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "07bef252cb524eaab2b3b0ce656ace0a"
format_stamp: "Formatted at 2024-05-03 14:00:38 on dist-test-slave-6w8v"
I20240503 14:00:38.657238 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:38.658236 2560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.658565 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:38.658638 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "07bef252cb524eaab2b3b0ce656ace0a"
format_stamp: "Formatted at 2024-05-03 14:00:38 on dist-test-slave-6w8v"
I20240503 14:00:38.658712 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:38.667976 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:38.668360 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:38.672298 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:43825
I20240503 14:00:38.672863 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:43825 every 8 connection(s)
I20240503 14:00:38.673384 2613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:38.675163 2613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a: Bootstrap starting.
I20240503 14:00:38.675988 2613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:38.676985 2613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a: No bootstrap required, opened a new log
I20240503 14:00:38.677356 2613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bef252cb524eaab2b3b0ce656ace0a" member_type: VOTER }
I20240503 14:00:38.677446 2613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:38.677475 2613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07bef252cb524eaab2b3b0ce656ace0a, State: Initialized, Role: FOLLOWER
I20240503 14:00:38.677577 2613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bef252cb524eaab2b3b0ce656ace0a" member_type: VOTER }
I20240503 14:00:38.677634 2613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:00:38.677659 2613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:00:38.677691 2613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:38.678344 2613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bef252cb524eaab2b3b0ce656ace0a" member_type: VOTER }
I20240503 14:00:38.678455 2613 leader_election.cc:302] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07bef252cb524eaab2b3b0ce656ace0a; no voters:
I20240503 14:00:38.678601 2613 leader_election.cc:288] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:00:38.678726 2616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:38.678908 2616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 1 LEADER]: Becoming Leader. State: Replica: 07bef252cb524eaab2b3b0ce656ace0a, State: Running, Role: LEADER
I20240503 14:00:38.679040 2616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bef252cb524eaab2b3b0ce656ace0a" member_type: VOTER }
I20240503 14:00:38.679073 2613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:00:38.679471 2618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07bef252cb524eaab2b3b0ce656ace0a. Latest consensus state: current_term: 1 leader_uuid: "07bef252cb524eaab2b3b0ce656ace0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bef252cb524eaab2b3b0ce656ace0a" member_type: VOTER } }
I20240503 14:00:38.679448 2617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07bef252cb524eaab2b3b0ce656ace0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07bef252cb524eaab2b3b0ce656ace0a" member_type: VOTER } }
I20240503 14:00:38.679674 2618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:38.679764 2617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:38.680123 2623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:00:38.680876 2623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:00:38.681082 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:00:38.682730 2623 catalog_manager.cc:1347] Generated new cluster ID: ac78315d35f84b7c9ebfa7781c3bd13a
I20240503 14:00:38.682782 2623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:00:38.702569 2623 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:00:38.703146 2623 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:00:38.710634 2623 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a: Generated new TSK 0
I20240503 14:00:38.710824 2623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:00:38.713410 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:38.716147 2635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:38.716940 2637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:38.717569 2634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:38.718122 396 server_base.cc:1034] running on GCE node
I20240503 14:00:38.718290 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:38.718331 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:38.718346 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744838718346 us; error 0 us; skew 500 ppm
I20240503 14:00:38.719272 396 webserver.cc:460] Webserver started at http://127.0.99.1:46873/ using document root <none> and password file <none>
I20240503 14:00:38.719422 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:38.719470 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:38.719554 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:38.719986 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-0-root/instance:
uuid: "3ec3c45adfd8480dba9c7c87fa5994df"
format_stamp: "Formatted at 2024-05-03 14:00:38 on dist-test-slave-6w8v"
I20240503 14:00:38.721411 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:38.722282 2642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.722503 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:38.722569 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-0-root
uuid: "3ec3c45adfd8480dba9c7c87fa5994df"
format_stamp: "Formatted at 2024-05-03 14:00:38 on dist-test-slave-6w8v"
I20240503 14:00:38.722640 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:38.758031 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:38.758477 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:38.758841 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:38.759346 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:38.759385 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.759438 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:38.759470 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.763959 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:35935
I20240503 14:00:38.763988 2705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:35935 every 8 connection(s)
I20240503 14:00:38.764806 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:38.767390 2710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:38.768218 2711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:38.768985 396 server_base.cc:1034] running on GCE node
I20240503 14:00:38.769418 2706 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43825
I20240503 14:00:38.769515 2706 heartbeater.cc:464] Registering TS with master...
W20240503 14:00:38.769629 2713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:38.769783 2706 heartbeater.cc:511] Master 127.0.99.62:43825 requested a full tablet report, sending...
I20240503 14:00:38.769876 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:38.769919 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:38.769945 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744838769944 us; error 0 us; skew 500 ppm
I20240503 14:00:38.770563 2577 ts_manager.cc:194] Registered new tserver with Master: 3ec3c45adfd8480dba9c7c87fa5994df (127.0.99.1:35935)
I20240503 14:00:38.771004 396 webserver.cc:460] Webserver started at http://127.0.99.2:38867/ using document root <none> and password file <none>
I20240503 14:00:38.771152 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:38.771203 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:38.771274 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:38.771499 2577 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54990
I20240503 14:00:38.771745 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-1-root/instance:
uuid: "84e1306418bf46e2869e44d39f329138"
format_stamp: "Formatted at 2024-05-03 14:00:38 on dist-test-slave-6w8v"
I20240503 14:00:38.773408 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:38.774343 2718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.774591 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:38.774658 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-1-root
uuid: "84e1306418bf46e2869e44d39f329138"
format_stamp: "Formatted at 2024-05-03 14:00:38 on dist-test-slave-6w8v"
I20240503 14:00:38.774734 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:38.781855 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:38.782271 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:38.782558 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:38.783031 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:38.783071 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.783115 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:38.783142 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.787364 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:45487
I20240503 14:00:38.787416 2781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:45487 every 8 connection(s)
I20240503 14:00:38.788197 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:38.790865 2785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:38.791765 2786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:38.792749 2788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:38.792784 2782 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43825
I20240503 14:00:38.792891 2782 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:38.793164 2782 heartbeater.cc:511] Master 127.0.99.62:43825 requested a full tablet report, sending...
I20240503 14:00:38.793361 396 server_base.cc:1034] running on GCE node
I20240503 14:00:38.793532 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:38.793571 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:38.793591 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744838793591 us; error 0 us; skew 500 ppm
I20240503 14:00:38.793846 2577 ts_manager.cc:194] Registered new tserver with Master: 84e1306418bf46e2869e44d39f329138 (127.0.99.2:45487)
I20240503 14:00:38.794443 2577 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54994
I20240503 14:00:38.794680 396 webserver.cc:460] Webserver started at http://127.0.99.3:36011/ using document root <none> and password file <none>
I20240503 14:00:38.794860 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:38.794920 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:38.794999 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:38.795415 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-2-root/instance:
uuid: "654f57f366004c7b8fc39b11c08135e9"
format_stamp: "Formatted at 2024-05-03 14:00:38 on dist-test-slave-6w8v"
I20240503 14:00:38.796938 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:38.797883 2793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.798174 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:38.798244 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-2-root
uuid: "654f57f366004c7b8fc39b11c08135e9"
format_stamp: "Formatted at 2024-05-03 14:00:38 on dist-test-slave-6w8v"
I20240503 14:00:38.798318 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714744785267298-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:38.806056 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:38.806499 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:38.806808 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:38.807309 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:38.807351 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.807396 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:38.807422 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:38.812458 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:41181
I20240503 14:00:38.812567 2856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:41181 every 8 connection(s)
I20240503 14:00:38.817114 2857 heartbeater.cc:347] Connected to a master server at 127.0.99.62:43825
I20240503 14:00:38.817255 2857 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:38.817509 2857 heartbeater.cc:511] Master 127.0.99.62:43825 requested a full tablet report, sending...
I20240503 14:00:38.818135 2577 ts_manager.cc:194] Registered new tserver with Master: 654f57f366004c7b8fc39b11c08135e9 (127.0.99.3:41181)
I20240503 14:00:38.818528 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005636636s
I20240503 14:00:38.818836 2577 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55004
I20240503 14:00:39.772722 2706 heartbeater.cc:503] Master 127.0.99.62:43825 was elected leader, sending a full tablet report...
I20240503 14:00:39.795490 2782 heartbeater.cc:503] Master 127.0.99.62:43825 was elected leader, sending a full tablet report...
I20240503 14:00:39.819890 2857 heartbeater.cc:503] Master 127.0.99.62:43825 was elected leader, sending a full tablet report...
I20240503 14:00:40.681056 2631 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:42.681471 2631 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:44.681877 2631 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:46.682236 2631 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:48.682608 2631 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:50.683008 2631 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:50.897850 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 14:00:50.901371 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:00:50.913343 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 14:00:50.915252 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 14:00:50.918239 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:00:50.930024 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 14:00:50.931897 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 14:00:50.934696 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:00:50.946434 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 14:00:50.948199 396 master.cc:537] Master@127.0.99.62:43825 shutting down...
I20240503 14:00:50.950834 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:00:50.950981 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:00:50.951030 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 07bef252cb524eaab2b3b0ce656ace0a: stopping tablet replica
I20240503 14:00:50.963079 396 master.cc:552] Master@127.0.99.62:43825 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12319 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240503 14:00:50.968268 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:38803
I20240503 14:00:50.968627 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:50.971179 2864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:50.971954 2861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:50.972703 396 server_base.cc:1034] running on GCE node
W20240503 14:00:50.973183 2862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:50.973393 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:50.973440 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:50.973460 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744850973460 us; error 0 us; skew 500 ppm
I20240503 14:00:50.974423 396 webserver.cc:460] Webserver started at http://127.0.99.62:33911/ using document root <none> and password file <none>
I20240503 14:00:50.974573 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:50.974625 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:50.974705 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:50.975113 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e"
format_stamp: "Formatted at 2024-05-03 14:00:50 on dist-test-slave-6w8v"
I20240503 14:00:50.976644 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 14:00:50.977627 2869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:50.977869 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:50.977944 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e"
format_stamp: "Formatted at 2024-05-03 14:00:50 on dist-test-slave-6w8v"
I20240503 14:00:50.978016 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:50.990603 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:50.991026 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:50.995132 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:38803
I20240503 14:00:50.996291 2921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:38803 every 8 connection(s)
I20240503 14:00:50.996721 2922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:50.998486 2922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e: Bootstrap starting.
I20240503 14:00:50.999279 2922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:51.000300 2922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e: No bootstrap required, opened a new log
I20240503 14:00:51.000682 2922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e" member_type: VOTER }
I20240503 14:00:51.000772 2922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:51.000805 2922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffe3eab5e5045b3b23eaf1fb83a688e, State: Initialized, Role: FOLLOWER
I20240503 14:00:51.000905 2922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e" member_type: VOTER }
I20240503 14:00:51.000958 2922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:00:51.000979 2922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:00:51.001010 2922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:51.001682 2922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e" member_type: VOTER }
I20240503 14:00:51.001791 2922 leader_election.cc:302] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ffe3eab5e5045b3b23eaf1fb83a688e; no voters:
I20240503 14:00:51.001948 2922 leader_election.cc:288] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:00:51.002068 2925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:51.002260 2925 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 1 LEADER]: Becoming Leader. State: Replica: 7ffe3eab5e5045b3b23eaf1fb83a688e, State: Running, Role: LEADER
I20240503 14:00:51.002344 2922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:00:51.002393 2925 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e" member_type: VOTER }
I20240503 14:00:51.002792 2926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e" member_type: VOTER } }
I20240503 14:00:51.002898 2926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:51.002806 2927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ffe3eab5e5045b3b23eaf1fb83a688e. Latest consensus state: current_term: 1 leader_uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffe3eab5e5045b3b23eaf1fb83a688e" member_type: VOTER } }
I20240503 14:00:51.003158 2930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:00:51.003165 2927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [sys.catalog]: This master's current role is: LEADER
I20240503 14:00:51.004047 2930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:00:51.004253 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:00:51.005852 2930 catalog_manager.cc:1347] Generated new cluster ID: 3dc39fe4122c4892a83f0806c5c44391
I20240503 14:00:51.005913 2930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:00:51.024945 2930 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:00:51.025575 2930 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:00:51.035692 2930 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e: Generated new TSK 0
I20240503 14:00:51.035910 2930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:00:51.068938 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:51.071758 2943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:51.072172 2944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:51.072777 396 server_base.cc:1034] running on GCE node
W20240503 14:00:51.073572 2946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:51.073833 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:51.073884 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:51.073899 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744851073899 us; error 0 us; skew 500 ppm
I20240503 14:00:51.074944 396 webserver.cc:460] Webserver started at http://127.0.99.1:42665/ using document root <none> and password file <none>
I20240503 14:00:51.075100 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:51.075145 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:51.075217 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:51.075690 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-0-root/instance:
uuid: "dd71aa2a46bf45f79d95b88d610c6cd8"
format_stamp: "Formatted at 2024-05-03 14:00:51 on dist-test-slave-6w8v"
I20240503 14:00:51.077188 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:51.078260 2951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.078576 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:51.078646 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-0-root
uuid: "dd71aa2a46bf45f79d95b88d610c6cd8"
format_stamp: "Formatted at 2024-05-03 14:00:51 on dist-test-slave-6w8v"
I20240503 14:00:51.078722 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:51.099496 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:51.099970 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:51.100275 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:51.100778 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:51.100818 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.100860 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:51.100891 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.105302 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:37319
I20240503 14:00:51.105319 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:37319 every 8 connection(s)
I20240503 14:00:51.106202 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:51.108973 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:51.109525 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:51.110060 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:51.110945 396 server_base.cc:1034] running on GCE node
I20240503 14:00:51.111070 3015 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38803
I20240503 14:00:51.111166 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:00:51.111174 3015 heartbeater.cc:464] Registering TS with master...
W20240503 14:00:51.111218 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:51.111263 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744851111263 us; error 0 us; skew 500 ppm
I20240503 14:00:51.111516 3015 heartbeater.cc:511] Master 127.0.99.62:38803 requested a full tablet report, sending...
I20240503 14:00:51.112253 2886 ts_manager.cc:194] Registered new tserver with Master: dd71aa2a46bf45f79d95b88d610c6cd8 (127.0.99.1:37319)
I20240503 14:00:51.112438 396 webserver.cc:460] Webserver started at http://127.0.99.2:44037/ using document root <none> and password file <none>
I20240503 14:00:51.112610 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:51.112669 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:51.112746 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:51.113209 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-1-root/instance:
uuid: "aa3b0fdb83e9417aa5e984de6cc0cd98"
format_stamp: "Formatted at 2024-05-03 14:00:51 on dist-test-slave-6w8v"
I20240503 14:00:51.113377 2886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56280
I20240503 14:00:51.115052 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:00:51.116145 3027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.116456 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 14:00:51.116531 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-1-root
uuid: "aa3b0fdb83e9417aa5e984de6cc0cd98"
format_stamp: "Formatted at 2024-05-03 14:00:51 on dist-test-slave-6w8v"
I20240503 14:00:51.116606 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:51.135339 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:51.135816 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:51.136119 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:51.136612 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:51.136653 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.136698 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:51.136726 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.140950 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:33333
I20240503 14:00:51.141047 3090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:33333 every 8 connection(s)
I20240503 14:00:51.141795 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:51.144490 3094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:51.144982 3095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:51.145507 3097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:51.146014 3091 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38803
I20240503 14:00:51.146111 3091 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:51.146268 396 server_base.cc:1034] running on GCE node
I20240503 14:00:51.146468 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240503 14:00:51.146509 3091 heartbeater.cc:511] Master 127.0.99.62:38803 requested a full tablet report, sending...
W20240503 14:00:51.146509 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:51.146597 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744851146596 us; error 0 us; skew 500 ppm
I20240503 14:00:51.147409 2886 ts_manager.cc:194] Registered new tserver with Master: aa3b0fdb83e9417aa5e984de6cc0cd98 (127.0.99.2:33333)
I20240503 14:00:51.147781 396 webserver.cc:460] Webserver started at http://127.0.99.3:45971/ using document root <none> and password file <none>
I20240503 14:00:51.147955 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:51.148012 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:51.148079 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:51.148181 2886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56292
I20240503 14:00:51.148542 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-2-root/instance:
uuid: "0817ca0f74484d9eae0f21d0520e1074"
format_stamp: "Formatted at 2024-05-03 14:00:51 on dist-test-slave-6w8v"
I20240503 14:00:51.150045 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 14:00:51.151197 3102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.151551 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:00:51.151647 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-2-root
uuid: "0817ca0f74484d9eae0f21d0520e1074"
format_stamp: "Formatted at 2024-05-03 14:00:51 on dist-test-slave-6w8v"
I20240503 14:00:51.151719 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:51.186973 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:51.187469 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:51.187830 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:51.188397 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:51.188439 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.188475 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:51.188503 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.193095 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:33393
I20240503 14:00:51.193234 3165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:33393 every 8 connection(s)
I20240503 14:00:51.193931 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:00:51.196662 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:51.197103 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:00:51.197784 3172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:00:51.198134 3166 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38803
I20240503 14:00:51.198238 3166 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:51.198500 3166 heartbeater.cc:511] Master 127.0.99.62:38803 requested a full tablet report, sending...
I20240503 14:00:51.198601 396 server_base.cc:1034] running on GCE node
I20240503 14:00:51.198832 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:00:51.198920 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:00:51.198952 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744851198951 us; error 0 us; skew 500 ppm
I20240503 14:00:51.199182 2886 ts_manager.cc:194] Registered new tserver with Master: 0817ca0f74484d9eae0f21d0520e1074 (127.0.99.3:33393)
I20240503 14:00:51.199776 2886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56300
I20240503 14:00:51.200134 396 webserver.cc:460] Webserver started at http://127.0.99.4:35939/ using document root <none> and password file <none>
I20240503 14:00:51.200299 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:00:51.200350 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:00:51.200414 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:00:51.200819 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-3-root/instance:
uuid: "6e6293a4c1964799b49d4b3e248ebfd9"
format_stamp: "Formatted at 2024-05-03 14:00:51 on dist-test-slave-6w8v"
I20240503 14:00:51.202358 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 14:00:51.203435 3177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.203801 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:51.203879 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-3-root
uuid: "6e6293a4c1964799b49d4b3e248ebfd9"
format_stamp: "Formatted at 2024-05-03 14:00:51 on dist-test-slave-6w8v"
I20240503 14:00:51.203963 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714744785267298-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:00:51.214195 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:00:51.214679 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:00:51.215014 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:00:51.215524 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:00:51.215606 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.215655 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:00:51.215683 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:00:51.220333 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:32919
I20240503 14:00:51.220394 3240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:32919 every 8 connection(s)
I20240503 14:00:51.225097 3241 heartbeater.cc:347] Connected to a master server at 127.0.99.62:38803
I20240503 14:00:51.225219 3241 heartbeater.cc:464] Registering TS with master...
I20240503 14:00:51.225510 3241 heartbeater.cc:511] Master 127.0.99.62:38803 requested a full tablet report, sending...
I20240503 14:00:51.226109 2886 ts_manager.cc:194] Registered new tserver with Master: 6e6293a4c1964799b49d4b3e248ebfd9 (127.0.99.4:32919)
I20240503 14:00:51.226471 396 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.005676834s
I20240503 14:00:51.226738 2886 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56314
I20240503 14:00:52.114526 3015 heartbeater.cc:503] Master 127.0.99.62:38803 was elected leader, sending a full tablet report...
I20240503 14:00:52.149309 3091 heartbeater.cc:503] Master 127.0.99.62:38803 was elected leader, sending a full tablet report...
I20240503 14:00:52.200830 3166 heartbeater.cc:503] Master 127.0.99.62:38803 was elected leader, sending a full tablet report...
I20240503 14:00:52.227942 3241 heartbeater.cc:503] Master 127.0.99.62:38803 was elected leader, sending a full tablet report...
I20240503 14:00:52.258337 396 test_util.cc:268] Using random seed: 634166813
I20240503 14:00:52.266505 2886 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56338:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240503 14:00:52.277037 2979 tablet_service.cc:1467] Processing CreateTablet for tablet 0d278c74fb60435883860d8f9ecf6717 (DEFAULT_TABLE table=test-workload [id=584c0b6ebf614404aff0c04fdfbb93c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:52.277153 3205 tablet_service.cc:1467] Processing CreateTablet for tablet 0d278c74fb60435883860d8f9ecf6717 (DEFAULT_TABLE table=test-workload [id=584c0b6ebf614404aff0c04fdfbb93c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:52.277402 3205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d278c74fb60435883860d8f9ecf6717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:52.277402 2979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d278c74fb60435883860d8f9ecf6717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:52.277621 3130 tablet_service.cc:1467] Processing CreateTablet for tablet 0d278c74fb60435883860d8f9ecf6717 (DEFAULT_TABLE table=test-workload [id=584c0b6ebf614404aff0c04fdfbb93c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:00:52.277905 3130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d278c74fb60435883860d8f9ecf6717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:00:52.279981 3263 tablet_bootstrap.cc:492] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9: Bootstrap starting.
I20240503 14:00:52.280324 3262 tablet_bootstrap.cc:492] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074: Bootstrap starting.
I20240503 14:00:52.281033 3263 tablet_bootstrap.cc:654] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:52.281154 3262 tablet_bootstrap.cc:654] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:52.281777 3264 tablet_bootstrap.cc:492] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8: Bootstrap starting.
I20240503 14:00:52.282343 3262 tablet_bootstrap.cc:492] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074: No bootstrap required, opened a new log
I20240503 14:00:52.282406 3263 tablet_bootstrap.cc:492] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9: No bootstrap required, opened a new log
I20240503 14:00:52.282478 3262 ts_tablet_manager.cc:1367] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:00:52.282541 3263 ts_tablet_manager.cc:1367] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:00:52.282701 3264 tablet_bootstrap.cc:654] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8: Neither blocks nor log segments found. Creating new log.
I20240503 14:00:52.282999 3262 raft_consensus.cc:348] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.283020 3263 raft_consensus.cc:348] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.283136 3262 raft_consensus.cc:374] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:52.283150 3263 raft_consensus.cc:374] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:52.283212 3262 raft_consensus.cc:729] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0817ca0f74484d9eae0f21d0520e1074, State: Initialized, Role: FOLLOWER
I20240503 14:00:52.283267 3263 raft_consensus.cc:729] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6293a4c1964799b49d4b3e248ebfd9, State: Initialized, Role: FOLLOWER
I20240503 14:00:52.283466 3263 consensus_queue.cc:260] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.283466 3262 consensus_queue.cc:260] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.283892 3264 tablet_bootstrap.cc:492] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8: No bootstrap required, opened a new log
I20240503 14:00:52.283921 3262 ts_tablet_manager.cc:1398] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:52.283926 3263 ts_tablet_manager.cc:1398] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:00:52.283983 3264 ts_tablet_manager.cc:1367] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:00:52.284374 3264 raft_consensus.cc:348] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.284478 3264 raft_consensus.cc:374] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:00:52.284510 3264 raft_consensus.cc:729] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd71aa2a46bf45f79d95b88d610c6cd8, State: Initialized, Role: FOLLOWER
I20240503 14:00:52.284626 3264 consensus_queue.cc:260] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.284962 3264 ts_tablet_manager.cc:1398] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20240503 14:00:52.357095 3016 tablet.cc:2369] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:00:52.447365 3270 raft_consensus.cc:482] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:00:52.447527 3270 raft_consensus.cc:504] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.448127 3270 leader_election.cc:288] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0817ca0f74484d9eae0f21d0520e1074 (127.0.99.3:33393), 6e6293a4c1964799b49d4b3e248ebfd9 (127.0.99.4:32919)
I20240503 14:00:52.451685 3140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d278c74fb60435883860d8f9ecf6717" candidate_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0817ca0f74484d9eae0f21d0520e1074" is_pre_election: true
I20240503 14:00:52.451689 3215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d278c74fb60435883860d8f9ecf6717" candidate_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" is_pre_election: true
I20240503 14:00:52.451857 3140 raft_consensus.cc:2454] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd71aa2a46bf45f79d95b88d610c6cd8 in term 0.
I20240503 14:00:52.451885 3215 raft_consensus.cc:2454] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd71aa2a46bf45f79d95b88d610c6cd8 in term 0.
I20240503 14:00:52.452203 2955 leader_election.cc:302] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0817ca0f74484d9eae0f21d0520e1074, 6e6293a4c1964799b49d4b3e248ebfd9, dd71aa2a46bf45f79d95b88d610c6cd8; no voters:
I20240503 14:00:52.452400 3270 raft_consensus.cc:2781] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:00:52.452463 3270 raft_consensus.cc:482] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:00:52.452503 3270 raft_consensus.cc:3037] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:52.453413 3270 raft_consensus.cc:504] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.453794 3270 leader_election.cc:288] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [CANDIDATE]: Term 1 election: Requested vote from peers 0817ca0f74484d9eae0f21d0520e1074 (127.0.99.3:33393), 6e6293a4c1964799b49d4b3e248ebfd9 (127.0.99.4:32919)
I20240503 14:00:52.454113 3215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d278c74fb60435883860d8f9ecf6717" candidate_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e6293a4c1964799b49d4b3e248ebfd9"
I20240503 14:00:52.454210 3215 raft_consensus.cc:3037] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:52.455042 3215 raft_consensus.cc:2454] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd71aa2a46bf45f79d95b88d610c6cd8 in term 1.
I20240503 14:00:52.455274 3140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d278c74fb60435883860d8f9ecf6717" candidate_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0817ca0f74484d9eae0f21d0520e1074"
I20240503 14:00:52.455317 2955 leader_election.cc:302] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e6293a4c1964799b49d4b3e248ebfd9, dd71aa2a46bf45f79d95b88d610c6cd8; no voters:
I20240503 14:00:52.455376 3140 raft_consensus.cc:3037] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:00:52.455456 3270 raft_consensus.cc:2781] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:00:52.455706 3270 raft_consensus.cc:686] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 1 LEADER]: Becoming Leader. State: Replica: dd71aa2a46bf45f79d95b88d610c6cd8, State: Running, Role: LEADER
I20240503 14:00:52.455889 3270 consensus_queue.cc:237] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } }
I20240503 14:00:52.456394 3140 raft_consensus.cc:2454] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd71aa2a46bf45f79d95b88d610c6cd8 in term 1.
I20240503 14:00:52.457476 2885 catalog_manager.cc:5484] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd71aa2a46bf45f79d95b88d610c6cd8 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd71aa2a46bf45f79d95b88d610c6cd8" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 37319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 } health_report { overall_health: UNKNOWN } } }
I20240503 14:00:52.952719 3270 consensus_queue.cc:1035] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e6293a4c1964799b49d4b3e248ebfd9" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:52.955718 3278 consensus_queue.cc:1035] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0817ca0f74484d9eae0f21d0520e1074" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 33393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:00:53.004343 2940 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:53.004638 2940 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:53.004710 2940 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:55.005028 2940 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:55.005331 2940 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:55.005383 2940 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:57.005722 2940 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:57.006080 2940 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:57.006152 2940 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:00:59.006481 2940 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:00:59.006817 2940 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:00:59.006915 2940 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:01.007264 2940 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:01.007653 2940 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:01:01.007725 2940 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:03.008126 2940 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:03.008486 2940 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:01:03.008559 2940 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:03.513377 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 14:01:03.516973 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:01:03.517190 396 tablet_replica.cc:330] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8: stopping tablet replica
I20240503 14:01:03.517321 396 raft_consensus.cc:2229] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:01:03.517546 396 raft_consensus.cc:2258] T 0d278c74fb60435883860d8f9ecf6717 P dd71aa2a46bf45f79d95b88d610c6cd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:03.530534 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 14:01:03.532927 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 14:01:03.536355 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:01:03.548512 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 14:01:03.550612 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 14:01:03.554327 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:01:03.554543 396 tablet_replica.cc:330] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074: stopping tablet replica
I20240503 14:01:03.554679 396 raft_consensus.cc:2229] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:03.554813 396 raft_consensus.cc:2258] T 0d278c74fb60435883860d8f9ecf6717 P 0817ca0f74484d9eae0f21d0520e1074 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:03.568007 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 14:01:03.570305 396 tablet_server.cc:178] TabletServer@127.0.99.4:0 shutting down...
I20240503 14:01:03.573925 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:01:03.574156 396 tablet_replica.cc:330] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9: stopping tablet replica
I20240503 14:01:03.574286 396 raft_consensus.cc:2229] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:03.574421 396 raft_consensus.cc:2258] T 0d278c74fb60435883860d8f9ecf6717 P 6e6293a4c1964799b49d4b3e248ebfd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:03.587206 396 tablet_server.cc:194] TabletServer@127.0.99.4:0 shutdown complete.
I20240503 14:01:03.589596 396 master.cc:537] Master@127.0.99.62:38803 shutting down...
I20240503 14:01:03.592552 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:01:03.592744 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:03.592795 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7ffe3eab5e5045b3b23eaf1fb83a688e: stopping tablet replica
I20240503 14:01:03.605262 396 master.cc:552] Master@127.0.99.62:38803 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12643 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240503 14:01:03.612269 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:44165
I20240503 14:01:03.612660 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:03.615828 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:03.616701 396 server_base.cc:1034] running on GCE node
W20240503 14:01:03.617337 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:03.617867 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:03.618124 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:03.618176 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:03.618196 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744863618196 us; error 0 us; skew 500 ppm
I20240503 14:01:03.625875 396 webserver.cc:460] Webserver started at http://127.0.99.62:33151/ using document root <none> and password file <none>
I20240503 14:01:03.626123 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:03.626183 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:03.626273 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:03.626739 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "37b0062345a84960898cd760055a3e1d"
format_stamp: "Formatted at 2024-05-03 14:01:03 on dist-test-slave-6w8v"
I20240503 14:01:03.628422 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 14:01:03.629451 3292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.629760 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:03.629838 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "37b0062345a84960898cd760055a3e1d"
format_stamp: "Formatted at 2024-05-03 14:01:03 on dist-test-slave-6w8v"
I20240503 14:01:03.629918 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:03.658988 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:03.659416 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:03.663897 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:44165
I20240503 14:01:03.665691 3344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:44165 every 8 connection(s)
I20240503 14:01:03.666159 3345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:03.668191 3345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d: Bootstrap starting.
I20240503 14:01:03.669040 3345 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:03.670190 3345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d: No bootstrap required, opened a new log
I20240503 14:01:03.670631 3345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b0062345a84960898cd760055a3e1d" member_type: VOTER }
I20240503 14:01:03.670732 3345 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:03.670753 3345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37b0062345a84960898cd760055a3e1d, State: Initialized, Role: FOLLOWER
I20240503 14:01:03.670882 3345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b0062345a84960898cd760055a3e1d" member_type: VOTER }
I20240503 14:01:03.670943 3345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:01:03.670979 3345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:01:03.671023 3345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:03.671859 3345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b0062345a84960898cd760055a3e1d" member_type: VOTER }
I20240503 14:01:03.671990 3345 leader_election.cc:302] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37b0062345a84960898cd760055a3e1d; no voters:
I20240503 14:01:03.672187 3345 leader_election.cc:288] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:01:03.672330 3348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:03.672526 3348 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 1 LEADER]: Becoming Leader. State: Replica: 37b0062345a84960898cd760055a3e1d, State: Running, Role: LEADER
I20240503 14:01:03.672672 3348 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b0062345a84960898cd760055a3e1d" member_type: VOTER }
I20240503 14:01:03.672732 3345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:01:03.673178 3349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37b0062345a84960898cd760055a3e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b0062345a84960898cd760055a3e1d" member_type: VOTER } }
I20240503 14:01:03.673221 3350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37b0062345a84960898cd760055a3e1d. Latest consensus state: current_term: 1 leader_uuid: "37b0062345a84960898cd760055a3e1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37b0062345a84960898cd760055a3e1d" member_type: VOTER } }
I20240503 14:01:03.673400 3350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [sys.catalog]: This master's current role is: LEADER
I20240503 14:01:03.673602 3349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [sys.catalog]: This master's current role is: LEADER
I20240503 14:01:03.673944 3355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:01:03.674726 3355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:01:03.674932 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:01:03.676908 3355 catalog_manager.cc:1347] Generated new cluster ID: f81141d098cb41c39369cebed4a8f38f
I20240503 14:01:03.676999 3355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:01:03.699448 3355 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:01:03.700122 3355 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:01:03.710708 3355 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d: Generated new TSK 0
I20240503 14:01:03.710940 3355 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:01:03.739907 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:03.742699 3366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:03.743252 3367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:03.744130 3369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:03.745038 396 server_base.cc:1034] running on GCE node
I20240503 14:01:03.745225 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:03.745272 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:03.745301 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744863745300 us; error 0 us; skew 500 ppm
I20240503 14:01:03.746315 396 webserver.cc:460] Webserver started at http://127.0.99.1:32987/ using document root <none> and password file <none>
I20240503 14:01:03.746479 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:03.746536 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:03.746634 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:03.747107 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-0-root/instance:
uuid: "5be8d3f03cce411faeb124d138bb4957"
format_stamp: "Formatted at 2024-05-03 14:01:03 on dist-test-slave-6w8v"
I20240503 14:01:03.749028 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:01:03.750128 3374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.750424 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:03.750494 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-0-root
uuid: "5be8d3f03cce411faeb124d138bb4957"
format_stamp: "Formatted at 2024-05-03 14:01:03 on dist-test-slave-6w8v"
I20240503 14:01:03.750566 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:03.789798 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:03.790215 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:03.790489 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:03.790995 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:03.791036 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.791177 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:03.791245 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.795702 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:44639
I20240503 14:01:03.795732 3437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:44639 every 8 connection(s)
I20240503 14:01:03.796625 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:03.799396 3442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:03.799901 3443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:03.800525 3445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:03.801182 3438 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44165
I20240503 14:01:03.801282 3438 heartbeater.cc:464] Registering TS with master...
I20240503 14:01:03.801332 396 server_base.cc:1034] running on GCE node
I20240503 14:01:03.801518 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:03.801564 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:03.801582 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744863801581 us; error 0 us; skew 500 ppm
I20240503 14:01:03.801599 3438 heartbeater.cc:511] Master 127.0.99.62:44165 requested a full tablet report, sending...
I20240503 14:01:03.802346 3309 ts_manager.cc:194] Registered new tserver with Master: 5be8d3f03cce411faeb124d138bb4957 (127.0.99.1:44639)
I20240503 14:01:03.802757 396 webserver.cc:460] Webserver started at http://127.0.99.2:36535/ using document root <none> and password file <none>
I20240503 14:01:03.802922 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:03.802982 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:03.803061 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:03.803392 3309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40754
I20240503 14:01:03.803635 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-1-root/instance:
uuid: "4b10b384e3dc4aae8623cf68ef91958e"
format_stamp: "Formatted at 2024-05-03 14:01:03 on dist-test-slave-6w8v"
I20240503 14:01:03.805213 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:03.806182 3450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.806407 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:03.806474 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-1-root
uuid: "4b10b384e3dc4aae8623cf68ef91958e"
format_stamp: "Formatted at 2024-05-03 14:01:03 on dist-test-slave-6w8v"
I20240503 14:01:03.806560 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:03.833683 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:03.834159 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:03.834476 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:03.835019 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:03.835060 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.835115 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:03.835142 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.839385 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:38185
I20240503 14:01:03.839527 3513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:38185 every 8 connection(s)
I20240503 14:01:03.840749 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:03.843544 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:03.844141 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:03.844924 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:03.845410 3514 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44165
I20240503 14:01:03.845513 3514 heartbeater.cc:464] Registering TS with master...
I20240503 14:01:03.845700 396 server_base.cc:1034] running on GCE node
I20240503 14:01:03.845764 3514 heartbeater.cc:511] Master 127.0.99.62:44165 requested a full tablet report, sending...
I20240503 14:01:03.845913 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:03.845968 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:03.845988 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744863845987 us; error 0 us; skew 500 ppm
I20240503 14:01:03.846449 3309 ts_manager.cc:194] Registered new tserver with Master: 4b10b384e3dc4aae8623cf68ef91958e (127.0.99.2:38185)
I20240503 14:01:03.847220 396 webserver.cc:460] Webserver started at http://127.0.99.3:35695/ using document root <none> and password file <none>
I20240503 14:01:03.847260 3309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40756
I20240503 14:01:03.847393 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:03.847455 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:03.847534 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:03.847997 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-2-root/instance:
uuid: "e4b4719de9a44356849520583cdc8f0a"
format_stamp: "Formatted at 2024-05-03 14:01:03 on dist-test-slave-6w8v"
I20240503 14:01:03.849464 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240503 14:01:03.850416 3525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.850765 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:03.850840 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-2-root
uuid: "e4b4719de9a44356849520583cdc8f0a"
format_stamp: "Formatted at 2024-05-03 14:01:03 on dist-test-slave-6w8v"
I20240503 14:01:03.850909 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714744785267298-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:03.876719 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:03.877148 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:03.877434 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:03.877938 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:03.877979 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.878012 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:03.878036 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:03.882856 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:43773
I20240503 14:01:03.882910 3588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:43773 every 8 connection(s)
I20240503 14:01:03.887748 3589 heartbeater.cc:347] Connected to a master server at 127.0.99.62:44165
I20240503 14:01:03.887878 3589 heartbeater.cc:464] Registering TS with master...
I20240503 14:01:03.888142 3589 heartbeater.cc:511] Master 127.0.99.62:44165 requested a full tablet report, sending...
I20240503 14:01:03.888803 3309 ts_manager.cc:194] Registered new tserver with Master: e4b4719de9a44356849520583cdc8f0a (127.0.99.3:43773)
I20240503 14:01:03.889034 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005735292s
I20240503 14:01:03.889390 3309 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40762
I20240503 14:01:04.804563 3438 heartbeater.cc:503] Master 127.0.99.62:44165 was elected leader, sending a full tablet report...
I20240503 14:01:04.848348 3514 heartbeater.cc:503] Master 127.0.99.62:44165 was elected leader, sending a full tablet report...
I20240503 14:01:04.890434 3589 heartbeater.cc:503] Master 127.0.99.62:44165 was elected leader, sending a full tablet report...
I20240503 14:01:04.921244 396 test_util.cc:268] Using random seed: 646829721
I20240503 14:01:04.929915 3309 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40792:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:01:04.930470 3309 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:01:04.941126 3553 tablet_service.cc:1467] Processing CreateTablet for tablet ece7163e95f44b8e8589edf8de74b9de (DEFAULT_TABLE table=test-workload [id=a261393e05b74e8583e1cc6e178cb7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:01:04.941126 3478 tablet_service.cc:1467] Processing CreateTablet for tablet ece7163e95f44b8e8589edf8de74b9de (DEFAULT_TABLE table=test-workload [id=a261393e05b74e8583e1cc6e178cb7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:01:04.941561 3553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece7163e95f44b8e8589edf8de74b9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:04.941583 3478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece7163e95f44b8e8589edf8de74b9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:04.941777 3402 tablet_service.cc:1467] Processing CreateTablet for tablet ece7163e95f44b8e8589edf8de74b9de (DEFAULT_TABLE table=test-workload [id=a261393e05b74e8583e1cc6e178cb7cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:01:04.942013 3402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece7163e95f44b8e8589edf8de74b9de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:04.944049 3610 tablet_bootstrap.cc:492] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e: Bootstrap starting.
I20240503 14:01:04.944665 3612 tablet_bootstrap.cc:492] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957: Bootstrap starting.
I20240503 14:01:04.944703 3611 tablet_bootstrap.cc:492] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a: Bootstrap starting.
I20240503 14:01:04.945302 3610 tablet_bootstrap.cc:654] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:04.945641 3612 tablet_bootstrap.cc:654] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:04.945645 3611 tablet_bootstrap.cc:654] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:04.946769 3610 tablet_bootstrap.cc:492] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e: No bootstrap required, opened a new log
I20240503 14:01:04.946873 3610 ts_tablet_manager.cc:1367] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:01:04.946893 3611 tablet_bootstrap.cc:492] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a: No bootstrap required, opened a new log
I20240503 14:01:04.946975 3611 ts_tablet_manager.cc:1367] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:01:04.947050 3612 tablet_bootstrap.cc:492] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957: No bootstrap required, opened a new log
I20240503 14:01:04.947127 3612 ts_tablet_manager.cc:1367] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:01:04.947379 3610 raft_consensus.cc:348] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:04.947487 3610 raft_consensus.cc:374] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:04.947520 3610 raft_consensus.cc:729] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b10b384e3dc4aae8623cf68ef91958e, State: Initialized, Role: FOLLOWER
I20240503 14:01:04.947463 3611 raft_consensus.cc:348] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:04.947501 3612 raft_consensus.cc:348] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:04.947657 3611 raft_consensus.cc:374] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:04.947679 3612 raft_consensus.cc:374] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:04.947719 3611 raft_consensus.cc:729] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b4719de9a44356849520583cdc8f0a, State: Initialized, Role: FOLLOWER
I20240503 14:01:04.947769 3612 raft_consensus.cc:729] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5be8d3f03cce411faeb124d138bb4957, State: Initialized, Role: FOLLOWER
I20240503 14:01:04.947750 3610 consensus_queue.cc:260] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:04.947870 3611 consensus_queue.cc:260] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:04.947984 3612 consensus_queue.cc:260] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:04.948283 3610 ts_tablet_manager.cc:1398] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:01:04.948305 3612 ts_tablet_manager.cc:1398] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:04.948267 3611 ts_tablet_manager.cc:1398] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:05.147792 3617 raft_consensus.cc:482] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:05.147956 3617 raft_consensus.cc:504] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:05.148746 3617 leader_election.cc:288] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4b4719de9a44356849520583cdc8f0a (127.0.99.3:43773), 5be8d3f03cce411faeb124d138bb4957 (127.0.99.1:44639)
I20240503 14:01:05.152762 3412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece7163e95f44b8e8589edf8de74b9de" candidate_uuid: "4b10b384e3dc4aae8623cf68ef91958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be8d3f03cce411faeb124d138bb4957" is_pre_election: true
I20240503 14:01:05.152853 3563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece7163e95f44b8e8589edf8de74b9de" candidate_uuid: "4b10b384e3dc4aae8623cf68ef91958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b4719de9a44356849520583cdc8f0a" is_pre_election: true
I20240503 14:01:05.152971 3412 raft_consensus.cc:2454] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b10b384e3dc4aae8623cf68ef91958e in term 0.
I20240503 14:01:05.152974 3563 raft_consensus.cc:2454] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b10b384e3dc4aae8623cf68ef91958e in term 0.
I20240503 14:01:05.153354 3451 leader_election.cc:302] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b10b384e3dc4aae8623cf68ef91958e, 5be8d3f03cce411faeb124d138bb4957; no voters:
I20240503 14:01:05.153532 3617 raft_consensus.cc:2781] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:05.153591 3617 raft_consensus.cc:482] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:05.153617 3617 raft_consensus.cc:3037] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:05.154592 3617 raft_consensus.cc:504] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:05.154997 3617 leader_election.cc:288] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [CANDIDATE]: Term 1 election: Requested vote from peers e4b4719de9a44356849520583cdc8f0a (127.0.99.3:43773), 5be8d3f03cce411faeb124d138bb4957 (127.0.99.1:44639)
I20240503 14:01:05.155403 3563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece7163e95f44b8e8589edf8de74b9de" candidate_uuid: "4b10b384e3dc4aae8623cf68ef91958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4b4719de9a44356849520583cdc8f0a"
I20240503 14:01:05.155428 3412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ece7163e95f44b8e8589edf8de74b9de" candidate_uuid: "4b10b384e3dc4aae8623cf68ef91958e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5be8d3f03cce411faeb124d138bb4957"
I20240503 14:01:05.155524 3563 raft_consensus.cc:3037] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:05.155524 3412 raft_consensus.cc:3037] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:05.156564 3563 raft_consensus.cc:2454] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b10b384e3dc4aae8623cf68ef91958e in term 1.
I20240503 14:01:05.156564 3412 raft_consensus.cc:2454] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b10b384e3dc4aae8623cf68ef91958e in term 1.
I20240503 14:01:05.156934 3451 leader_election.cc:302] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b10b384e3dc4aae8623cf68ef91958e, 5be8d3f03cce411faeb124d138bb4957; no voters:
I20240503 14:01:05.157114 3617 raft_consensus.cc:2781] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:05.157341 3617 raft_consensus.cc:686] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 1 LEADER]: Becoming Leader. State: Replica: 4b10b384e3dc4aae8623cf68ef91958e, State: Running, Role: LEADER
I20240503 14:01:05.157486 3617 consensus_queue.cc:237] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } }
I20240503 14:01:05.159170 3307 catalog_manager.cc:5484] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b10b384e3dc4aae8623cf68ef91958e (127.0.99.2). New cstate: current_term: 1 leader_uuid: "4b10b384e3dc4aae8623cf68ef91958e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b10b384e3dc4aae8623cf68ef91958e" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 38185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:05.541915 3617 consensus_queue.cc:1035] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5be8d3f03cce411faeb124d138bb4957" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 44639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:05.545105 3621 consensus_queue.cc:1035] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b4719de9a44356849520583cdc8f0a" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:05.674758 3363 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:05.675091 3363 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:01:05.675153 3363 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:05.675683 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:06.677493 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:07.675514 3363 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:07.675869 3363 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:01:07.675920 3363 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:07.679080 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:08.680778 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:09.676276 3363 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:09.676610 3363 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:01:09.676668 3363 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:09.682478 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:10.684242 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:11.676995 3363 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:11.677301 3363 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:01:11.677352 3363 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:11.685930 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:12.687660 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:13.677675 3363 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:13.678001 3363 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:01:13.678057 3363 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:13.689395 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:14.690981 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:15.678390 3363 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:15.678712 3363 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240503 14:01:15.678763 3363 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:15.692680 3357 placement_policy_util.cc:407] tablet ece7163e95f44b8e8589edf8de74b9de: detected majority of replicas (2 of 3) at location L0
I20240503 14:01:16.225067 396 tablet_server.cc:178] TabletServer@127.0.99.1:0 shutting down...
I20240503 14:01:16.228749 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:01:16.229012 396 tablet_replica.cc:330] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957: stopping tablet replica
I20240503 14:01:16.229135 396 raft_consensus.cc:2229] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:16.229277 396 raft_consensus.cc:2258] T ece7163e95f44b8e8589edf8de74b9de P 5be8d3f03cce411faeb124d138bb4957 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:16.242317 396 tablet_server.cc:194] TabletServer@127.0.99.1:0 shutdown complete.
I20240503 14:01:16.244732 396 tablet_server.cc:178] TabletServer@127.0.99.2:0 shutting down...
I20240503 14:01:16.248525 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:01:16.248736 396 tablet_replica.cc:330] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e: stopping tablet replica
I20240503 14:01:16.248860 396 raft_consensus.cc:2229] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:01:16.249071 396 raft_consensus.cc:2258] T ece7163e95f44b8e8589edf8de74b9de P 4b10b384e3dc4aae8623cf68ef91958e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:16.261893 396 tablet_server.cc:194] TabletServer@127.0.99.2:0 shutdown complete.
I20240503 14:01:16.263998 396 tablet_server.cc:178] TabletServer@127.0.99.3:0 shutting down...
I20240503 14:01:16.267328 396 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240503 14:01:16.267625 396 tablet_replica.cc:330] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a: stopping tablet replica
I20240503 14:01:16.267776 396 raft_consensus.cc:2229] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:16.267911 396 raft_consensus.cc:2258] T ece7163e95f44b8e8589edf8de74b9de P e4b4719de9a44356849520583cdc8f0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:16.280967 396 tablet_server.cc:194] TabletServer@127.0.99.3:0 shutdown complete.
I20240503 14:01:16.283100 396 master.cc:537] Master@127.0.99.62:44165 shutting down...
I20240503 14:01:16.286080 396 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:01:16.286294 396 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:16.286360 396 tablet_replica.cc:330] T 00000000000000000000000000000000 P 37b0062345a84960898cd760055a3e1d: stopping tablet replica
I20240503 14:01:16.299014 396 master.cc:552] Master@127.0.99.62:44165 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12693 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240503 14:01:16.306224 396 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.99.62:35165
I20240503 14:01:16.306603 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:16.309417 3637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:16.310180 3639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:16.311008 396 server_base.cc:1034] running on GCE node
W20240503 14:01:16.311483 3636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:16.311746 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:16.311793 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:16.311808 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744876311808 us; error 0 us; skew 500 ppm
I20240503 14:01:16.318794 396 webserver.cc:460] Webserver started at http://127.0.99.62:33245/ using document root <none> and password file <none>
I20240503 14:01:16.318977 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:16.319025 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:16.319093 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:16.319494 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/master-0-root/instance:
uuid: "92e573faf0e8433c8fac5c264373a791"
format_stamp: "Formatted at 2024-05-03 14:01:16 on dist-test-slave-6w8v"
I20240503 14:01:16.321111 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:16.322166 3644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.322465 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:16.322542 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/master-0-root
uuid: "92e573faf0e8433c8fac5c264373a791"
format_stamp: "Formatted at 2024-05-03 14:01:16 on dist-test-slave-6w8v"
I20240503 14:01:16.322630 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:16.329267 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:16.329792 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:16.334029 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.62:35165
I20240503 14:01:16.335675 3696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.62:35165 every 8 connection(s)
I20240503 14:01:16.336150 3697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:16.338141 3697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791: Bootstrap starting.
I20240503 14:01:16.339031 3697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:16.340278 3697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791: No bootstrap required, opened a new log
I20240503 14:01:16.340736 3697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e573faf0e8433c8fac5c264373a791" member_type: VOTER }
I20240503 14:01:16.340842 3697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:16.340871 3697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e573faf0e8433c8fac5c264373a791, State: Initialized, Role: FOLLOWER
I20240503 14:01:16.341002 3697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e573faf0e8433c8fac5c264373a791" member_type: VOTER }
I20240503 14:01:16.341066 3697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:01:16.341104 3697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:01:16.341152 3697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:16.341941 3697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e573faf0e8433c8fac5c264373a791" member_type: VOTER }
I20240503 14:01:16.342078 3697 leader_election.cc:302] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e573faf0e8433c8fac5c264373a791; no voters:
I20240503 14:01:16.342274 3697 leader_election.cc:288] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:01:16.342417 3700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:16.342679 3700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [term 1 LEADER]: Becoming Leader. State: Replica: 92e573faf0e8433c8fac5c264373a791, State: Running, Role: LEADER
I20240503 14:01:16.342801 3697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:01:16.342836 3700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e573faf0e8433c8fac5c264373a791" member_type: VOTER }
I20240503 14:01:16.343328 3701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92e573faf0e8433c8fac5c264373a791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e573faf0e8433c8fac5c264373a791" member_type: VOTER } }
I20240503 14:01:16.343372 3702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e573faf0e8433c8fac5c264373a791. Latest consensus state: current_term: 1 leader_uuid: "92e573faf0e8433c8fac5c264373a791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e573faf0e8433c8fac5c264373a791" member_type: VOTER } }
I20240503 14:01:16.343537 3702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [sys.catalog]: This master's current role is: LEADER
I20240503 14:01:16.343858 3701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791 [sys.catalog]: This master's current role is: LEADER
I20240503 14:01:16.344183 3707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:01:16.344915 3707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:01:16.345108 396 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240503 14:01:16.346940 3707 catalog_manager.cc:1347] Generated new cluster ID: ba46402bfee34faf8b09aeba792d3e67
I20240503 14:01:16.347002 3707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:01:16.362660 3707 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:01:16.363319 3707 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:01:16.369702 3707 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 92e573faf0e8433c8fac5c264373a791: Generated new TSK 0
I20240503 14:01:16.369947 3707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:01:16.377951 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:16.381098 3721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:16.381677 396 server_base.cc:1034] running on GCE node
W20240503 14:01:16.382472 3718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:16.383136 3719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:16.383350 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:16.383409 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:16.383430 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744876383430 us; error 0 us; skew 500 ppm
I20240503 14:01:16.384507 396 webserver.cc:460] Webserver started at http://127.0.99.1:40641/ using document root <none> and password file <none>
I20240503 14:01:16.384706 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:16.384765 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:16.384848 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:16.385257 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root/instance:
uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
format_stamp: "Formatted at 2024-05-03 14:01:16 on dist-test-slave-6w8v"
I20240503 14:01:16.386885 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 14:01:16.388067 3726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.388381 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:16.388464 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root
uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
format_stamp: "Formatted at 2024-05-03 14:01:16 on dist-test-slave-6w8v"
I20240503 14:01:16.388550 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:16.420171 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:16.420627 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:16.420940 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:16.421451 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:16.421492 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.421536 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:16.421564 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.426015 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.1:38053
I20240503 14:01:16.426040 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.1:38053 every 8 connection(s)
I20240503 14:01:16.426986 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:16.430022 3795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:16.430941 3797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:16.431416 3794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:16.431924 3790 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35165
I20240503 14:01:16.431939 396 server_base.cc:1034] running on GCE node
I20240503 14:01:16.432065 3790 heartbeater.cc:464] Registering TS with master...
I20240503 14:01:16.432137 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:16.432176 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:16.432199 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744876432198 us; error 0 us; skew 500 ppm
I20240503 14:01:16.432391 3790 heartbeater.cc:511] Master 127.0.99.62:35165 requested a full tablet report, sending...
I20240503 14:01:16.433138 3661 ts_manager.cc:194] Registered new tserver with Master: f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053)
I20240503 14:01:16.433961 3661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39640
I20240503 14:01:16.434803 396 webserver.cc:460] Webserver started at http://127.0.99.2:34629/ using document root <none> and password file <none>
I20240503 14:01:16.434964 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:16.435021 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:16.435106 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:16.435550 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root/instance:
uuid: "2d8cd9c1e8e2469888742770004dcb11"
format_stamp: "Formatted at 2024-05-03 14:01:16 on dist-test-slave-6w8v"
I20240503 14:01:16.437505 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:01:16.438740 3802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.439057 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:16.439131 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root
uuid: "2d8cd9c1e8e2469888742770004dcb11"
format_stamp: "Formatted at 2024-05-03 14:01:16 on dist-test-slave-6w8v"
I20240503 14:01:16.439211 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:16.446548 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:16.446981 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:16.447284 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:16.447799 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:16.447841 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.447885 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:16.447911 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.452107 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.2:40655
I20240503 14:01:16.452155 3865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.2:40655 every 8 connection(s)
I20240503 14:01:16.452991 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:16.455737 3870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:16.456193 3869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:16.456702 3872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:16.457255 3866 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35165
I20240503 14:01:16.457368 3866 heartbeater.cc:464] Registering TS with master...
I20240503 14:01:16.457538 396 server_base.cc:1034] running on GCE node
I20240503 14:01:16.457624 3866 heartbeater.cc:511] Master 127.0.99.62:35165 requested a full tablet report, sending...
I20240503 14:01:16.457743 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:16.457791 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:16.457806 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744876457806 us; error 0 us; skew 500 ppm
I20240503 14:01:16.458309 3661 ts_manager.cc:194] Registered new tserver with Master: 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:16.458966 3661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39644
I20240503 14:01:16.459055 396 webserver.cc:460] Webserver started at http://127.0.99.3:39937/ using document root <none> and password file <none>
I20240503 14:01:16.459213 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:16.459266 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:16.459370 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:16.459834 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root/instance:
uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
format_stamp: "Formatted at 2024-05-03 14:01:16 on dist-test-slave-6w8v"
I20240503 14:01:16.461305 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:16.462249 3877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.462494 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:16.462561 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root
uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
format_stamp: "Formatted at 2024-05-03 14:01:16 on dist-test-slave-6w8v"
I20240503 14:01:16.462636 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:16.477948 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:16.478458 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:16.478780 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:16.479279 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:16.479317 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.479363 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:16.479390 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:16.484125 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.3:43425
I20240503 14:01:16.484167 3940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.3:43425 every 8 connection(s)
I20240503 14:01:16.489044 3941 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35165
I20240503 14:01:16.489182 3941 heartbeater.cc:464] Registering TS with master...
I20240503 14:01:16.489462 3941 heartbeater.cc:511] Master 127.0.99.62:35165 requested a full tablet report, sending...
I20240503 14:01:16.490165 3661 ts_manager.cc:194] Registered new tserver with Master: f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:16.490288 396 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005709503s
I20240503 14:01:16.490753 3661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39650
I20240503 14:01:17.435124 3790 heartbeater.cc:503] Master 127.0.99.62:35165 was elected leader, sending a full tablet report...
I20240503 14:01:17.459995 3866 heartbeater.cc:503] Master 127.0.99.62:35165 was elected leader, sending a full tablet report...
I20240503 14:01:17.491832 3941 heartbeater.cc:503] Master 127.0.99.62:35165 was elected leader, sending a full tablet report...
I20240503 14:01:17.517259 396 test_util.cc:268] Using random seed: 659425736
I20240503 14:01:17.525943 3661 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39654:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240503 14:01:17.526611 3661 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240503 14:01:17.541385 3830 tablet_service.cc:1467] Processing CreateTablet for tablet d121433c26cb4d89876ce434ba669006 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 14:01:17.541623 3900 tablet_service.cc:1467] Processing CreateTablet for tablet 71dd0142015e4fb791413d533b9ccb55 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 14:01:17.541779 3830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d121433c26cb4d89876ce434ba669006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.541803 3904 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7539be4b0c41d1accce62f9abbc9e9 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 14:01:17.541910 3900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71dd0142015e4fb791413d533b9ccb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.542016 3904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b7539be4b0c41d1accce62f9abbc9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.542694 3903 tablet_service.cc:1467] Processing CreateTablet for tablet 8af6f6a1be8f43a693011cfa1c022f08 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 14:01:17.542932 3903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af6f6a1be8f43a693011cfa1c022f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.544777 3900 tablet_service.cc:1467] Processing CreateTablet for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 14:01:17.544785 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7539be4b0c41d1accce62f9abbc9e9 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 14:01:17.544991 3828 tablet_service.cc:1467] Processing CreateTablet for tablet 8af6f6a1be8f43a693011cfa1c022f08 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 14:01:17.545040 3829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b7539be4b0c41d1accce62f9abbc9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.545053 3902 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6ad905b31541ffa6f2007d886b7643 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 14:01:17.545225 3828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af6f6a1be8f43a693011cfa1c022f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.545040 3900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bde81d4ddf4de3a2e16be46d97b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.545432 3901 tablet_service.cc:1467] Processing CreateTablet for tablet 0171f05f97a24f97ab8732925b360c82 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 14:01:17.545646 3901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0171f05f97a24f97ab8732925b360c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.546886 3899 tablet_service.cc:1467] Processing CreateTablet for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 14:01:17.547132 3899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.547901 3902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6ad905b31541ffa6f2007d886b7643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.541635 3905 tablet_service.cc:1467] Processing CreateTablet for tablet d121433c26cb4d89876ce434ba669006 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 14:01:17.548789 3827 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6ad905b31541ffa6f2007d886b7643 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 14:01:17.548892 3905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d121433c26cb4d89876ce434ba669006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.549036 3827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6ad905b31541ffa6f2007d886b7643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.549209 3897 tablet_service.cc:1467] Processing CreateTablet for tablet cc6eba6dbb2c4efda6d24f9ee9d6e8f7 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 14:01:17.549445 3897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6eba6dbb2c4efda6d24f9ee9d6e8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.551205 3827 tablet_service.cc:1467] Processing CreateTablet for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 14:01:17.551477 3827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.551590 3825 tablet_service.cc:1467] Processing CreateTablet for tablet 71dd0142015e4fb791413d533b9ccb55 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 14:01:17.551831 3825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71dd0142015e4fb791413d533b9ccb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.551792 3824 tablet_service.cc:1467] Processing CreateTablet for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 14:01:17.552028 3824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.552158 3822 tablet_service.cc:1467] Processing CreateTablet for tablet cc6eba6dbb2c4efda6d24f9ee9d6e8f7 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 14:01:17.552381 3822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6eba6dbb2c4efda6d24f9ee9d6e8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.553072 3905 tablet_service.cc:1467] Processing CreateTablet for tablet 11893d5b97174d7389005a373091703b (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 14:01:17.553275 3905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11893d5b97174d7389005a373091703b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.554075 3898 tablet_service.cc:1467] Processing CreateTablet for tablet 11c78a8dc746409aa487dcd7ab58cc82 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 14:01:17.554292 3821 tablet_service.cc:1467] Processing CreateTablet for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 14:01:17.554298 3898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c78a8dc746409aa487dcd7ab58cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.554525 3821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bde81d4ddf4de3a2e16be46d97b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.555071 3899 tablet_service.cc:1467] Processing CreateTablet for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 14:01:17.555281 3899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.556162 3823 tablet_service.cc:1467] Processing CreateTablet for tablet 11c78a8dc746409aa487dcd7ab58cc82 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 14:01:17.556354 3823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c78a8dc746409aa487dcd7ab58cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.557227 3962 tablet_bootstrap.cc:492] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.557492 3754 tablet_service.cc:1467] Processing CreateTablet for tablet d121433c26cb4d89876ce434ba669006 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240503 14:01:17.557710 3754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d121433c26cb4d89876ce434ba669006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.558079 3753 tablet_service.cc:1467] Processing CreateTablet for tablet 6b7539be4b0c41d1accce62f9abbc9e9 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240503 14:01:17.558197 3826 tablet_service.cc:1467] Processing CreateTablet for tablet 0171f05f97a24f97ab8732925b360c82 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 14:01:17.558310 3753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b7539be4b0c41d1accce62f9abbc9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.558383 3826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0171f05f97a24f97ab8732925b360c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.558539 3962 tablet_bootstrap.cc:654] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.559756 3752 tablet_service.cc:1467] Processing CreateTablet for tablet 8af6f6a1be8f43a693011cfa1c022f08 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240503 14:01:17.560001 3752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af6f6a1be8f43a693011cfa1c022f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.559944 3754 tablet_service.cc:1467] Processing CreateTablet for tablet 11893d5b97174d7389005a373091703b (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 14:01:17.559955 3745 tablet_service.cc:1467] Processing CreateTablet for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240503 14:01:17.560155 3746 tablet_service.cc:1467] Processing CreateTablet for tablet cc6eba6dbb2c4efda6d24f9ee9d6e8f7 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240503 14:01:17.560369 3747 tablet_service.cc:1467] Processing CreateTablet for tablet 11c78a8dc746409aa487dcd7ab58cc82 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240503 14:01:17.560163 3754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11893d5b97174d7389005a373091703b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.560639 3748 tablet_service.cc:1467] Processing CreateTablet for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240503 14:01:17.560847 3748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.560993 3747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c78a8dc746409aa487dcd7ab58cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.561223 3749 tablet_service.cc:1467] Processing CreateTablet for tablet 71dd0142015e4fb791413d533b9ccb55 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240503 14:01:17.561425 3749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71dd0142015e4fb791413d533b9ccb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.561554 3753 tablet_service.cc:1467] Processing CreateTablet for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240503 14:01:17.561769 3753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.563019 3745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bde81d4ddf4de3a2e16be46d97b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.563290 3750 tablet_service.cc:1467] Processing CreateTablet for tablet 0171f05f97a24f97ab8732925b360c82 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240503 14:01:17.563534 3750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0171f05f97a24f97ab8732925b360c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.564105 3746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc6eba6dbb2c4efda6d24f9ee9d6e8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.564738 3751 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6ad905b31541ffa6f2007d886b7643 (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240503 14:01:17.565302 3751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6ad905b31541ffa6f2007d886b7643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.567595 3962 tablet_bootstrap.cc:492] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.567713 3962 ts_tablet_manager.cc:1367] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:01:17.568217 3962 raft_consensus.cc:348] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.568336 3962 raft_consensus.cc:374] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.568372 3962 raft_consensus.cc:729] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.568491 3962 consensus_queue.cc:260] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.568869 3962 ts_tablet_manager.cc:1398] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.568846 3828 tablet_service.cc:1467] Processing CreateTablet for tablet 11893d5b97174d7389005a373091703b (DEFAULT_TABLE table=test-workload [id=841e0752055246b7a9dbb6fb916cdab6]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240503 14:01:17.569094 3962 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.569237 3828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11893d5b97174d7389005a373091703b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:17.570093 3962 tablet_bootstrap.cc:654] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.571249 3962 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.571331 3962 ts_tablet_manager.cc:1367] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:01:17.571830 3962 raft_consensus.cc:348] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.572069 3962 raft_consensus.cc:374] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.572134 3962 raft_consensus.cc:729] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.572254 3962 consensus_queue.cc:260] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.572723 3962 ts_tablet_manager.cc:1398] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:01:17.573020 3961 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.573060 3962 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.574018 3961 tablet_bootstrap.cc:654] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.574280 3962 tablet_bootstrap.cc:654] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.575192 3964 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.576143 3964 tablet_bootstrap.cc:654] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.576609 3961 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.576695 3961 ts_tablet_manager.cc:1367] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:01:17.576774 3962 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.576866 3962 ts_tablet_manager.cc:1367] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:01:17.577114 3961 raft_consensus.cc:348] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.577205 3961 raft_consensus.cc:374] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.577236 3961 raft_consensus.cc:729] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.577270 3962 raft_consensus.cc:348] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.577371 3962 raft_consensus.cc:374] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.577419 3962 raft_consensus.cc:729] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.577378 3961 consensus_queue.cc:260] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.577536 3962 consensus_queue.cc:260] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.577751 3961 ts_tablet_manager.cc:1398] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.577867 3962 ts_tablet_manager.cc:1398] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.577924 3961 tablet_bootstrap.cc:492] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.578078 3962 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.579000 3962 tablet_bootstrap.cc:654] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.579005 3961 tablet_bootstrap.cc:654] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.580077 3961 tablet_bootstrap.cc:492] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.580096 3964 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.580173 3964 ts_tablet_manager.cc:1367] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:01:17.580163 3961 ts_tablet_manager.cc:1367] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:01:17.580682 3964 raft_consensus.cc:348] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.580804 3964 raft_consensus.cc:374] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.580848 3964 raft_consensus.cc:729] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.580966 3964 consensus_queue.cc:260] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.581300 3964 ts_tablet_manager.cc:1398] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.581537 3964 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.581558 3961 raft_consensus.cc:348] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.581768 3961 raft_consensus.cc:374] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.581812 3961 raft_consensus.cc:729] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.581930 3961 consensus_queue.cc:260] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.582168 3961 ts_tablet_manager.cc:1398] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:01:17.582320 3961 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.583216 3961 tablet_bootstrap.cc:654] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.583247 3962 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.583338 3962 ts_tablet_manager.cc:1367] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:01:17.583835 3962 raft_consensus.cc:348] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.583915 3964 tablet_bootstrap.cc:654] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.584499 3962 raft_consensus.cc:374] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.584563 3962 raft_consensus.cc:729] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.585202 3964 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.585280 3964 ts_tablet_manager.cc:1367] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:01:17.585438 3961 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.585521 3961 ts_tablet_manager.cc:1367] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:01:17.585700 3964 raft_consensus.cc:348] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.585799 3964 raft_consensus.cc:374] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.585820 3964 raft_consensus.cc:729] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.585865 3961 raft_consensus.cc:348] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.585963 3961 raft_consensus.cc:374] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.585945 3964 consensus_queue.cc:260] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.586004 3961 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.586143 3961 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.586190 3964 ts_tablet_manager.cc:1398] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:17.586344 3964 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.586367 3961 ts_tablet_manager.cc:1398] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.586544 3961 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.587348 3961 tablet_bootstrap.cc:654] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.587498 3964 tablet_bootstrap.cc:654] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.588219 3962 consensus_queue.cc:260] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.588639 3962 ts_tablet_manager.cc:1398] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:01:17.588855 3962 tablet_bootstrap.cc:492] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.588938 3964 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.589017 3964 ts_tablet_manager.cc:1367] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:01:17.589413 3964 raft_consensus.cc:348] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.589511 3964 raft_consensus.cc:374] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.589543 3964 raft_consensus.cc:729] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.589648 3964 consensus_queue.cc:260] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.589918 3964 ts_tablet_manager.cc:1398] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.589980 3962 tablet_bootstrap.cc:654] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.590086 3964 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.590924 3964 tablet_bootstrap.cc:654] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.591038 3962 tablet_bootstrap.cc:492] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.591113 3962 ts_tablet_manager.cc:1367] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:01:17.591522 3962 raft_consensus.cc:348] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.591670 3962 raft_consensus.cc:374] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.591707 3962 raft_consensus.cc:729] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.591822 3962 consensus_queue.cc:260] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.592114 3962 ts_tablet_manager.cc:1398] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.592276 3962 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.592617 3961 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.592679 3961 ts_tablet_manager.cc:1367] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240503 14:01:17.592967 3961 raft_consensus.cc:348] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.593029 3961 raft_consensus.cc:374] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.593047 3961 raft_consensus.cc:729] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.593152 3962 tablet_bootstrap.cc:654] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.593137 3961 consensus_queue.cc:260] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.593354 3961 ts_tablet_manager.cc:1398] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:17.593480 3961 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.594250 3961 tablet_bootstrap.cc:654] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.594234 3962 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.594358 3962 ts_tablet_manager.cc:1367] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:01:17.594761 3964 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.594744 3962 raft_consensus.cc:348] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.594837 3962 raft_consensus.cc:374] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.594866 3962 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.594837 3964 ts_tablet_manager.cc:1367] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:01:17.594959 3962 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.595203 3962 ts_tablet_manager.cc:1398] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:17.595302 3964 raft_consensus.cc:348] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.595378 3962 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.595391 3964 raft_consensus.cc:374] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.595453 3964 raft_consensus.cc:729] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.595628 3964 consensus_queue.cc:260] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.595871 3964 ts_tablet_manager.cc:1398] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.596029 3964 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.596300 3962 tablet_bootstrap.cc:654] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.596894 3964 tablet_bootstrap.cc:654] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.597214 3962 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.597296 3962 ts_tablet_manager.cc:1367] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:01:17.597699 3962 raft_consensus.cc:348] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.597813 3962 raft_consensus.cc:374] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.597844 3962 raft_consensus.cc:729] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.597990 3962 consensus_queue.cc:260] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.598089 3964 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.598166 3964 ts_tablet_manager.cc:1367] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:01:17.598256 3962 ts_tablet_manager.cc:1398] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:17.598408 3962 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.598552 3964 raft_consensus.cc:348] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.598639 3964 raft_consensus.cc:374] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.598670 3964 raft_consensus.cc:729] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.598795 3964 consensus_queue.cc:260] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.599136 3964 ts_tablet_manager.cc:1398] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.599330 3962 tablet_bootstrap.cc:654] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.599356 3964 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.600234 3964 tablet_bootstrap.cc:654] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.601219 3964 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.601325 3964 ts_tablet_manager.cc:1367] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:01:17.601238 3961 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.601435 3961 ts_tablet_manager.cc:1367] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:01:17.601722 3964 raft_consensus.cc:348] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.601763 3961 raft_consensus.cc:348] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.601811 3964 raft_consensus.cc:374] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.601833 3961 raft_consensus.cc:374] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.601841 3964 raft_consensus.cc:729] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.601862 3961 raft_consensus.cc:729] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.601956 3961 consensus_queue.cc:260] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.601962 3964 consensus_queue.cc:260] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.602207 3961 ts_tablet_manager.cc:1398] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:17.602259 3964 ts_tablet_manager.cc:1398] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:17.602368 3961 tablet_bootstrap.cc:492] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.602545 3964 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.602893 3962 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.602993 3962 ts_tablet_manager.cc:1367] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:01:17.603307 3961 tablet_bootstrap.cc:654] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.603307 3964 tablet_bootstrap.cc:654] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.604408 3962 raft_consensus.cc:348] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.604645 3962 raft_consensus.cc:374] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.604691 3962 raft_consensus.cc:729] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.604804 3962 consensus_queue.cc:260] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.605099 3962 ts_tablet_manager.cc:1398] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:01:17.605304 3962 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.605926 3961 tablet_bootstrap.cc:492] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.605932 3964 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.606029 3964 ts_tablet_manager.cc:1367] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:01:17.606029 3961 ts_tablet_manager.cc:1367] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:01:17.606276 3962 tablet_bootstrap.cc:654] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.606462 3961 raft_consensus.cc:348] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.606544 3961 raft_consensus.cc:374] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.606571 3961 raft_consensus.cc:729] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.606686 3961 consensus_queue.cc:260] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.606957 3961 ts_tablet_manager.cc:1398] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:17.607129 3964 raft_consensus.cc:348] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.607223 3964 raft_consensus.cc:374] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.607251 3964 raft_consensus.cc:729] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.607347 3964 consensus_queue.cc:260] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.607678 3964 ts_tablet_manager.cc:1398] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:01:17.607823 3964 tablet_bootstrap.cc:492] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.607409 3962 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.608428 3962 ts_tablet_manager.cc:1367] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:01:17.608733 3964 tablet_bootstrap.cc:654] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.608850 3962 raft_consensus.cc:348] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.608943 3962 raft_consensus.cc:374] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.608973 3962 raft_consensus.cc:729] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.609094 3962 consensus_queue.cc:260] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.609846 3962 ts_tablet_manager.cc:1398] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.610014 3962 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.607357 3961 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.611302 3961 tablet_bootstrap.cc:654] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.611302 3962 tablet_bootstrap.cc:654] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.612493 3961 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.612493 3962 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.612519 3964 tablet_bootstrap.cc:492] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.612605 3961 ts_tablet_manager.cc:1367] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:01:17.612701 3964 ts_tablet_manager.cc:1367] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:01:17.612701 3962 ts_tablet_manager.cc:1367] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:01:17.613085 3961 raft_consensus.cc:348] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.613193 3961 raft_consensus.cc:374] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.613240 3961 raft_consensus.cc:729] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.613162 3964 raft_consensus.cc:348] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.613162 3962 raft_consensus.cc:348] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.613377 3962 raft_consensus.cc:374] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.613380 3961 consensus_queue.cc:260] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.613369 3964 raft_consensus.cc:374] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.613476 3962 raft_consensus.cc:729] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.613499 3964 raft_consensus.cc:729] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.613596 3962 consensus_queue.cc:260] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.613623 3964 consensus_queue.cc:260] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.613704 3961 ts_tablet_manager.cc:1398] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.613860 3962 ts_tablet_manager.cc:1398] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.614023 3962 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.614282 3964 ts_tablet_manager.cc:1398] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:01:17.613883 3961 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.614434 3964 tablet_bootstrap.cc:492] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.614989 3962 tablet_bootstrap.cc:654] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.615236 3964 tablet_bootstrap.cc:654] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.615248 3961 tablet_bootstrap.cc:654] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.616916 3961 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.617014 3961 ts_tablet_manager.cc:1367] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:01:17.617514 3961 raft_consensus.cc:348] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.617578 3962 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.617636 3964 tablet_bootstrap.cc:492] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.617635 3961 raft_consensus.cc:374] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.617686 3962 ts_tablet_manager.cc:1367] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:01:17.617708 3961 raft_consensus.cc:729] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.617725 3964 ts_tablet_manager.cc:1367] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:01:17.617861 3961 consensus_queue.cc:260] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.618093 3962 raft_consensus.cc:348] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.618184 3961 ts_tablet_manager.cc:1398] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:17.618198 3962 raft_consensus.cc:374] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.618259 3962 raft_consensus.cc:729] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.618350 3961 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.618108 3964 raft_consensus.cc:348] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.618503 3964 raft_consensus.cc:374] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.618531 3964 raft_consensus.cc:729] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.618626 3964 consensus_queue.cc:260] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.618358 3962 consensus_queue.cc:260] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.618857 3964 ts_tablet_manager.cc:1398] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:17.618968 3962 ts_tablet_manager.cc:1398] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.619016 3964 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.619114 3962 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: Bootstrap starting.
I20240503 14:01:17.619396 3961 tablet_bootstrap.cc:654] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.619884 3964 tablet_bootstrap.cc:654] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.620173 3962 tablet_bootstrap.cc:654] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.620493 3961 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.620568 3961 ts_tablet_manager.cc:1367] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:01:17.621007 3961 raft_consensus.cc:348] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.621109 3961 raft_consensus.cc:374] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.621136 3961 raft_consensus.cc:729] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.621254 3961 consensus_queue.cc:260] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.621526 3961 ts_tablet_manager.cc:1398] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:17.621670 3964 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.621687 3961 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.621742 3964 ts_tablet_manager.cc:1367] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:01:17.621786 3962 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: No bootstrap required, opened a new log
I20240503 14:01:17.621847 3962 ts_tablet_manager.cc:1367] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:01:17.622089 3964 raft_consensus.cc:348] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.622162 3964 raft_consensus.cc:374] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.622181 3964 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.622262 3964 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.622299 3962 raft_consensus.cc:348] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.622390 3962 raft_consensus.cc:374] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.622421 3962 raft_consensus.cc:729] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.622488 3964 ts_tablet_manager.cc:1398] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:17.622548 3961 tablet_bootstrap.cc:654] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.622548 3962 consensus_queue.cc:260] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.622629 3964 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.623462 3964 tablet_bootstrap.cc:654] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.624641 3961 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.624724 3961 ts_tablet_manager.cc:1367] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:01:17.624781 3962 ts_tablet_manager.cc:1398] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:01:17.625078 3961 raft_consensus.cc:348] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.625162 3961 raft_consensus.cc:374] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.625190 3961 raft_consensus.cc:729] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.625303 3961 consensus_queue.cc:260] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.625579 3961 ts_tablet_manager.cc:1398] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:17.625732 3961 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.626590 3961 tablet_bootstrap.cc:654] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.627622 3961 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.627688 3964 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.627712 3961 ts_tablet_manager.cc:1367] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:01:17.627771 3964 ts_tablet_manager.cc:1367] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:01:17.628074 3961 raft_consensus.cc:348] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.628095 3964 raft_consensus.cc:348] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.628173 3961 raft_consensus.cc:374] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.628190 3964 raft_consensus.cc:374] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.628196 3961 raft_consensus.cc:729] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.628242 3964 raft_consensus.cc:729] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.628362 3961 consensus_queue.cc:260] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.628399 3964 consensus_queue.cc:260] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.628618 3961 ts_tablet_manager.cc:1398] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.628656 3964 ts_tablet_manager.cc:1398] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.628814 3964 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: Bootstrap starting.
I20240503 14:01:17.628822 3961 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: Bootstrap starting.
I20240503 14:01:17.629746 3964 tablet_bootstrap.cc:654] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.629753 3961 tablet_bootstrap.cc:654] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: Neither blocks nor log segments found. Creating new log.
I20240503 14:01:17.630972 3961 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: No bootstrap required, opened a new log
I20240503 14:01:17.631050 3961 ts_tablet_manager.cc:1367] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:01:17.631460 3961 raft_consensus.cc:348] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.631543 3964 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: No bootstrap required, opened a new log
I20240503 14:01:17.631587 3961 raft_consensus.cc:374] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.631616 3969 raft_consensus.cc:482] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.631690 3961 raft_consensus.cc:729] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.631705 3964 ts_tablet_manager.cc:1367] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:01:17.631736 3969 raft_consensus.cc:504] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.631805 3961 consensus_queue.cc:260] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.632174 3961 ts_tablet_manager.cc:1398] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.632128 3964 raft_consensus.cc:348] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.632244 3964 raft_consensus.cc:374] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:01:17.632282 3964 raft_consensus.cc:729] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Initialized, Role: FOLLOWER
I20240503 14:01:17.632385 3964 consensus_queue.cc:260] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.632645 3964 ts_tablet_manager.cc:1398] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:17.632247 3969 leader_election.cc:288] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.636960 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.637125 3764 raft_consensus.cc:2454] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.637434 3881 leader_election.cc:302] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.637650 4003 raft_consensus.cc:2781] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.637707 4003 raft_consensus.cc:482] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.637753 4003 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.637944 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11" is_pre_election: true
I20240503 14:01:17.638057 3840 raft_consensus.cc:2454] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.638739 4003 raft_consensus.cc:504] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.639073 4003 leader_election.cc:288] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.639413 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.639509 3764 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.639495 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:17.639662 3840 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.640358 4003 raft_consensus.cc:482] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.640417 3840 raft_consensus.cc:2454] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.640427 3764 raft_consensus.cc:2454] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.640699 3880 leader_election.cc:302] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.640904 3969 raft_consensus.cc:2781] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.641074 3969 raft_consensus.cc:686] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:17.641207 3969 consensus_queue.cc:237] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.640446 4003 raft_consensus.cc:504] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.642257 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11" is_pre_election: true
I20240503 14:01:17.642257 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.642395 3840 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.642427 3764 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.642516 4003 leader_election.cc:288] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.642632 3880 leader_election.cc:302] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:17.642766 4003 raft_consensus.cc:2781] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.642808 4003 raft_consensus.cc:482] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.642863 4003 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.642814 3659 catalog_manager.cc:5484] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } }
I20240503 14:01:17.643599 4003 raft_consensus.cc:504] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.643939 4003 leader_election.cc:288] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.644409 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.644464 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:17.644547 3840 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.644546 3764 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.645395 3764 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.645395 3840 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.645651 3881 leader_election.cc:302] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.645797 4003 raft_consensus.cc:2781] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.645884 4003 raft_consensus.cc:686] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:17.645998 4003 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.647387 3659 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } }
I20240503 14:01:17.657119 3973 raft_consensus.cc:482] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.657280 3973 raft_consensus.cc:504] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.657852 3973 leader_election.cc:288] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.661211 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d121433c26cb4d89876ce434ba669006" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11" is_pre_election: true
I20240503 14:01:17.661360 3840 raft_consensus.cc:2454] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 0.
I20240503 14:01:17.661624 3729 leader_election.cc:302] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.661799 3973 raft_consensus.cc:2781] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.661774 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d121433c26cb4d89876ce434ba669006" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" is_pre_election: true
I20240503 14:01:17.661859 3973 raft_consensus.cc:482] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.661927 3973 raft_consensus.cc:3037] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.661949 3915 raft_consensus.cc:2454] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 0.
I20240503 14:01:17.662946 3973 raft_consensus.cc:504] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.663241 3973 leader_election.cc:288] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.663519 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d121433c26cb4d89876ce434ba669006" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:17.663640 3840 raft_consensus.cc:3037] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.663597 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d121433c26cb4d89876ce434ba669006" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:17.663740 3915 raft_consensus.cc:3037] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.664429 3840 raft_consensus.cc:2454] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 1.
I20240503 14:01:17.664430 3915 raft_consensus.cc:2454] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 1.
I20240503 14:01:17.664690 3729 leader_election.cc:302] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.664765 3965 raft_consensus.cc:482] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.664848 3965 raft_consensus.cc:504] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.664928 3973 raft_consensus.cc:2781] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.665128 3973 raft_consensus.cc:686] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:17.665272 3965 leader_election.cc:288] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.665284 3973 consensus_queue.cc:237] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } }
I20240503 14:01:17.667796 3659 catalog_manager.cc:5484] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } }
I20240503 14:01:17.668684 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.668802 3764 raft_consensus.cc:2454] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.669250 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" is_pre_election: true
I20240503 14:01:17.669251 3806 leader_election.cc:302] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.669368 3915 raft_consensus.cc:2454] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.669443 3965 raft_consensus.cc:2781] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.669499 3965 raft_consensus.cc:482] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.669544 3965 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.670469 3965 raft_consensus.cc:504] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.670742 3965 leader_election.cc:288] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.671048 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:17.671088 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.671154 3915 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.671161 3764 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.671931 3764 raft_consensus.cc:2454] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.671938 3915 raft_consensus.cc:2454] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.672295 3805 leader_election.cc:302] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.672487 3965 raft_consensus.cc:2781] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.672678 3965 raft_consensus.cc:686] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:17.672809 3965 consensus_queue.cc:237] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.674304 3659 catalog_manager.cc:5484] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:17.675976 4003 raft_consensus.cc:482] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.676057 4003 raft_consensus.cc:504] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.676523 4003 leader_election.cc:288] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.676693 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.676770 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11" is_pre_election: true
I20240503 14:01:17.676831 3764 raft_consensus.cc:2454] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.676864 3840 raft_consensus.cc:2454] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.677076 3881 leader_election.cc:302] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.677229 4003 raft_consensus.cc:2781] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.677289 4003 raft_consensus.cc:482] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.677333 4003 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.678139 4003 raft_consensus.cc:504] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.678453 4003 leader_election.cc:288] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.678757 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:17.678849 3840 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.679553 3840 raft_consensus.cc:2454] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.679677 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.679782 3764 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.679843 3880 leader_election.cc:302] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:17.680001 4003 raft_consensus.cc:2781] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.680085 4003 raft_consensus.cc:686] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:17.680213 4003 consensus_queue.cc:237] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.680658 3764 raft_consensus.cc:2454] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.681883 3659 catalog_manager.cc:5484] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } }
I20240503 14:01:17.701359 4003 raft_consensus.cc:482] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.701522 4003 raft_consensus.cc:504] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.701931 4003 leader_election.cc:288] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.702399 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c78a8dc746409aa487dcd7ab58cc82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11" is_pre_election: true
I20240503 14:01:17.702395 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c78a8dc746409aa487dcd7ab58cc82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.702562 3840 raft_consensus.cc:2454] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.702602 3965 raft_consensus.cc:482] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.702567 3764 raft_consensus.cc:2454] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.702718 3965 raft_consensus.cc:504] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.702951 3880 leader_election.cc:302] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:17.703125 4003 raft_consensus.cc:2781] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.703181 4003 raft_consensus.cc:482] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.703229 4003 raft_consensus.cc:3037] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.703186 3965 leader_election.cc:288] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.703538 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7539be4b0c41d1accce62f9abbc9e9" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" is_pre_election: true
I20240503 14:01:17.703645 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7539be4b0c41d1accce62f9abbc9e9" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.703691 3915 raft_consensus.cc:2454] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.703727 3764 raft_consensus.cc:2454] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.703925 3806 leader_election.cc:302] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.704077 3965 raft_consensus.cc:2781] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.704133 3965 raft_consensus.cc:482] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.704169 3965 raft_consensus.cc:3037] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.704874 4003 raft_consensus.cc:504] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.705173 3965 raft_consensus.cc:504] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.705483 3965 leader_election.cc:288] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.705593 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c78a8dc746409aa487dcd7ab58cc82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.705675 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c78a8dc746409aa487dcd7ab58cc82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:17.705713 4003 leader_election.cc:288] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.705786 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7539be4b0c41d1accce62f9abbc9e9" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:17.705775 3763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7539be4b0c41d1accce62f9abbc9e9" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.705808 3840 raft_consensus.cc:3037] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.705890 3915 raft_consensus.cc:3037] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.705916 3763 raft_consensus.cc:3037] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.705694 3764 raft_consensus.cc:3037] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.706768 3915 raft_consensus.cc:2454] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.706789 3840 raft_consensus.cc:2454] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.707159 3880 leader_election.cc:302] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:17.707181 3805 leader_election.cc:302] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:17.707334 3965 raft_consensus.cc:2781] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.707322 3763 raft_consensus.cc:2454] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.707320 4003 raft_consensus.cc:2781] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.707475 4003 raft_consensus.cc:686] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:17.707652 4003 consensus_queue.cc:237] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.706768 3764 raft_consensus.cc:2454] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.707696 3965 raft_consensus.cc:686] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:17.707824 3965 consensus_queue.cc:237] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.709667 3659 catalog_manager.cc:5484] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:17.709769 3661 catalog_manager.cc:5484] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } }
I20240503 14:01:17.711023 3973 raft_consensus.cc:482] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.711117 3973 raft_consensus.cc:504] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.711475 3973 leader_election.cc:288] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.711999 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11893d5b97174d7389005a373091703b" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11" is_pre_election: true
I20240503 14:01:17.712047 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11893d5b97174d7389005a373091703b" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" is_pre_election: true
I20240503 14:01:17.712124 3840 raft_consensus.cc:2454] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 0.
I20240503 14:01:17.712160 3915 raft_consensus.cc:2454] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 0.
I20240503 14:01:17.712347 3729 leader_election.cc:302] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.712515 3973 raft_consensus.cc:2781] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.712565 3973 raft_consensus.cc:482] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.712591 3973 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.713346 3973 raft_consensus.cc:504] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.713642 3973 leader_election.cc:288] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 1 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.713943 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11893d5b97174d7389005a373091703b" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:17.713971 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11893d5b97174d7389005a373091703b" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:17.714082 3840 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.714092 3915 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.714988 3840 raft_consensus.cc:2454] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 1.
I20240503 14:01:17.714988 3915 raft_consensus.cc:2454] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 1.
I20240503 14:01:17.715299 3729 leader_election.cc:302] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.715435 3973 raft_consensus.cc:2781] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.715529 3973 raft_consensus.cc:686] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:17.715708 3973 consensus_queue.cc:237] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.717329 3661 catalog_manager.cc:5484] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 1 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:17.728013 4003 raft_consensus.cc:482] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.728190 4003 raft_consensus.cc:504] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.728660 4003 leader_election.cc:288] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.728933 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11" is_pre_election: true
I20240503 14:01:17.729049 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.729074 3840 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.729250 3764 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 0.
I20240503 14:01:17.729427 3880 leader_election.cc:302] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:17.729605 4003 raft_consensus.cc:2781] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.729666 4003 raft_consensus.cc:482] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.729693 4003 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.730507 4003 raft_consensus.cc:504] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.730844 4003 leader_election.cc:288] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:17.731205 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.731205 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:17.731352 3840 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.731352 3764 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.732337 3764 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.732337 3840 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 1.
I20240503 14:01:17.732676 3880 leader_election.cc:302] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:17.732841 4003 raft_consensus.cc:2781] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.732930 4003 raft_consensus.cc:686] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:17.733070 4003 consensus_queue.cc:237] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.734735 3661 catalog_manager.cc:5484] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } }
I20240503 14:01:17.742691 3965 raft_consensus.cc:482] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.742849 3965 raft_consensus.cc:504] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.743301 3965 leader_election.cc:288] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.743793 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.743793 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" is_pre_election: true
I20240503 14:01:17.743984 3764 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.743984 3915 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.744289 3806 leader_election.cc:302] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.744467 3965 raft_consensus.cc:2781] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.744529 3965 raft_consensus.cc:482] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.744566 3965 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.745437 3965 raft_consensus.cc:504] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.745767 3965 leader_election.cc:288] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.746098 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:17.746223 3915 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.746992 3915 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.747246 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.747291 3805 leader_election.cc:302] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:17.747354 3764 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.747462 3965 raft_consensus.cc:2781] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.747546 3965 raft_consensus.cc:686] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:17.747727 3965 consensus_queue.cc:237] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.748198 3764 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.749423 3661 catalog_manager.cc:5484] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:17.810249 3965 raft_consensus.cc:482] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.810431 3965 raft_consensus.cc:504] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.810874 3965 leader_election.cc:288] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.811338 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" is_pre_election: true
I20240503 14:01:17.811336 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.811497 3915 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.811503 3764 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.811833 3805 leader_election.cc:302] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.811990 3965 raft_consensus.cc:2781] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.812042 3965 raft_consensus.cc:482] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.812067 3965 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.812867 3965 raft_consensus.cc:504] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.813206 3965 leader_election.cc:288] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.813555 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.813562 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:17.813666 3764 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.813678 3915 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.814428 3915 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.814432 3764 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.814730 3806 leader_election.cc:302] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.814891 3965 raft_consensus.cc:2781] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.814985 3965 raft_consensus.cc:686] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:17.815126 3965 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.816865 3661 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:17.907582 3965 raft_consensus.cc:482] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:01:17.907745 3965 raft_consensus.cc:504] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.908149 3965 leader_election.cc:288] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.908573 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af6f6a1be8f43a693011cfa1c022f08" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" is_pre_election: true
I20240503 14:01:17.908576 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af6f6a1be8f43a693011cfa1c022f08" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" is_pre_election: true
I20240503 14:01:17.908704 3915 raft_consensus.cc:2454] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.908751 3764 raft_consensus.cc:2454] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 0.
I20240503 14:01:17.909000 3806 leader_election.cc:302] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.909169 3965 raft_consensus.cc:2781] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:01:17.909224 3965 raft_consensus.cc:482] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:01:17.909250 3965 raft_consensus.cc:3037] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.910061 3965 raft_consensus.cc:504] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.910516 3965 leader_election.cc:288] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:17.910849 3764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af6f6a1be8f43a693011cfa1c022f08" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:17.910856 3915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af6f6a1be8f43a693011cfa1c022f08" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:17.911010 3764 raft_consensus.cc:3037] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.911043 3915 raft_consensus.cc:3037] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:01:17.911783 3915 raft_consensus.cc:2454] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.911783 3764 raft_consensus.cc:2454] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 1.
I20240503 14:01:17.912114 3806 leader_election.cc:302] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:17.912341 3965 raft_consensus.cc:2781] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:01:17.912436 3965 raft_consensus.cc:686] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:17.912578 3965 consensus_queue.cc:237] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:17.914285 3661 catalog_manager.cc:5484] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:18.021579 3915 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.022347 3764 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.022579 4015 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:01:18.024710 3764 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.025068 3965 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.026188 3763 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.026557 3969 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.028235 4015 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.030126 3914 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.030205 3761 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.030632 3840 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.030700 4006 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.031059 3839 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.031383 3838 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.031925 3762 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.033077 4006 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.034865 4003 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.035161 4003 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.034919 4006 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.038805 3914 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.034588 4015 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.043963 3839 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.044435 3973 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.050030 4044 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240503 14:01:18.050488 3911 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.051465 3914 raft_consensus.cc:1261] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.051505 3759 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.056481 3913 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.057389 4044 consensus_queue.cc:1035] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:01:18.058429 3965 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.057755 3836 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:01:18.056545 3835 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:01:18.062513 3973 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.062671 3756 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.065104 3758 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:01:18.066975 3755 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:01:18.067227 3969 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:01:18.061548 3914 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:01:18.067555 3757 raft_consensus.cc:1261] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:01:18.067713 4056 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240503 14:01:18.074760 4015 consensus_queue.cc:1035] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.067875 4058 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20240503 14:01:18.068744 4003 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240503 14:01:18.075523 4063 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20240503 14:01:18.066980 3834 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:01:18.077718 3965 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240503 14:01:18.079217 4048 mvcc.cc:205] Tried to move back new op lower bound from 7023595020379017216 to 7023595019532230656. Current Snapshot: MvccSnapshot[applied={T|T < 7023595020379017216}]
I20240503 14:01:18.088896 4063 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:01:18.102648 4076 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:01:18.105443 4039 mvcc.cc:205] Tried to move back new op lower bound from 7023595020516954112 to 7023595019931447296. Current Snapshot: MvccSnapshot[applied={T|T < 7023595020449816576}]
I20240503 14:01:18.128301 4047 mvcc.cc:205] Tried to move back new op lower bound from 7023595020535963648 to 7023595019094720512. Current Snapshot: MvccSnapshot[applied={T|T < 7023595020460175360}]
I20240503 14:01:18.344908 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:18.401002 3912 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
mode: GRACEFUL
new_leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:47982
I20240503 14:01:18.401212 3912 raft_consensus.cc:595] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Received request to transfer leadership to TS f71f8fdeb4004bb8b8012dc8b79ff8e1
W20240503 14:01:18.410759 3886 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47978: Service unavailable: leader transfer in progress
W20240503 14:01:18.412814 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47978: Service unavailable: leader transfer in progress
I20240503 14:01:18.413254 4080 raft_consensus.cc:979] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87: : Instructing follower f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:18.413345 4080 raft_consensus.cc:1067] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Signalling peer f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:18.416651 3758 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:52334
I20240503 14:01:18.416817 3758 raft_consensus.cc:482] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:18.416900 3758 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:18.418606 3758 raft_consensus.cc:504] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:18.419152 3758 leader_election.cc:288] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
W20240503 14:01:18.420768 3886 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47978: Service unavailable: leader transfer in progress
I20240503 14:01:18.421392 3836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:18.421504 3836 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:18.422636 3836 raft_consensus.cc:2454] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:18.431867 3729 leader_election.cc:302] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:18.432197 3973 raft_consensus.cc:2781] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:18.432294 3973 raft_consensus.cc:686] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:18.432415 3973 consensus_queue.cc:237] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:18.433404 3834 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:18.433585 3834 raft_consensus.cc:595] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Received request to transfer leadership to TS f71f8fdeb4004bb8b8012dc8b79ff8e1
I20240503 14:01:18.434590 3657 catalog_manager.cc:5484] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 1 to 2, leader changed from f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:18.435040 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 14:01:18.435200 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:01:18.437680 3888 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47978: Service unavailable: leader transfer in progress
I20240503 14:01:18.445236 3912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:18.445395 3912 raft_consensus.cc:3032] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:18.445442 3912 raft_consensus.cc:729] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:18.445600 3912 consensus_queue.cc:260] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 1.19, Last appended by leader: 19, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:18.445762 3912 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:18.446810 3912 raft_consensus.cc:2454] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:18.458464 4108 raft_consensus.cc:979] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:18.458705 4100 raft_consensus.cc:1067] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Signalling peer f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:18.461189 3758 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:52346
I20240503 14:01:18.461369 3758 raft_consensus.cc:482] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:18.461441 3758 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:18.462728 3758 raft_consensus.cc:504] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:18.464280 3758 leader_election.cc:288] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:18.464787 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:18.464922 3837 raft_consensus.cc:3032] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:18.464982 3837 raft_consensus.cc:729] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:18.465130 3837 consensus_queue.cc:260] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:18.465297 3837 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:18.466647 3837 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:18.466974 3729 leader_election.cc:302] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:18.467409 4098 raft_consensus.cc:2781] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:18.467518 4098 raft_consensus.cc:686] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:18.467691 4098 consensus_queue.cc:237] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 23, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:18.469580 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:18.469717 3914 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:18.469836 3657 catalog_manager.cc:5484] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 1 to 2, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:18.471086 3914 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:18.471413 3913 raft_consensus.cc:1226] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Rejecting Update request from peer 2d8cd9c1e8e2469888742770004dcb11 for earlier term 1. Current term is 2. Ops: []
W20240503 14:01:18.473794 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47978: Illegal state: replica f6d3c1f17e6b4f42a813bef57d94fe87 is not leader of this config: current role FOLLOWER
W20240503 14:01:18.473963 3888 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47978: Illegal state: replica f6d3c1f17e6b4f42a813bef57d94fe87 is not leader of this config: current role FOLLOWER
W20240503 14:01:18.474510 3888 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47978: Illegal state: replica f6d3c1f17e6b4f42a813bef57d94fe87 is not leader of this config: current role FOLLOWER
W20240503 14:01:18.474488 3889 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47978: Illegal state: replica f6d3c1f17e6b4f42a813bef57d94fe87 is not leader of this config: current role FOLLOWER
I20240503 14:01:18.479597 3840 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20240503 14:01:18.480310 3973 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20240503 14:01:18.480662 3913 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 19. Preceding OpId from leader: term: 2 index: 21. (index mismatch)
I20240503 14:01:18.480978 3973 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20240503 14:01:18.509828 3820 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36476: Illegal state: replica 2d8cd9c1e8e2469888742770004dcb11 is not leader of this config: current role FOLLOWER
W20240503 14:01:18.509999 3818 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36476: Illegal state: replica 2d8cd9c1e8e2469888742770004dcb11 is not leader of this config: current role FOLLOWER
W20240503 14:01:18.516815 3814 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36476: Illegal state: replica 2d8cd9c1e8e2469888742770004dcb11 is not leader of this config: current role FOLLOWER
I20240503 14:01:18.539575 3837 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:01:18.540547 3911 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:01:18.540776 3973 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240503 14:01:18.542769 3973 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.000s
I20240503 14:01:18.555855 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:18.558930 4117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:18.559768 4114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:18.560194 396 server_base.cc:1034] running on GCE node
W20240503 14:01:18.561103 4115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:18.561431 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:18.561486 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:18.561517 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744878561517 us; error 0 us; skew 500 ppm
I20240503 14:01:18.562901 396 webserver.cc:460] Webserver started at http://127.0.99.4:40741/ using document root <none> and password file <none>
I20240503 14:01:18.563082 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:18.563140 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:18.563220 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:18.563818 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-3-root/instance:
uuid: "6d05d524f254418bb351065633a4606c"
format_stamp: "Formatted at 2024-05-03 14:01:18 on dist-test-slave-6w8v"
I20240503 14:01:18.565774 396 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240503 14:01:18.567060 4122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.567391 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240503 14:01:18.567474 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-3-root
uuid: "6d05d524f254418bb351065633a4606c"
format_stamp: "Formatted at 2024-05-03 14:01:18 on dist-test-slave-6w8v"
I20240503 14:01:18.567667 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:18.574286 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:18.574725 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:18.575066 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:18.575721 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:18.575762 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.575796 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:18.575824 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.584192 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.4:42471
I20240503 14:01:18.584235 4185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.4:42471 every 8 connection(s)
I20240503 14:01:18.585279 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:18.588771 4190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:18.589416 396 server_base.cc:1034] running on GCE node
I20240503 14:01:18.589690 4186 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35165
I20240503 14:01:18.589807 4186 heartbeater.cc:464] Registering TS with master...
I20240503 14:01:18.590119 4186 heartbeater.cc:511] Master 127.0.99.62:35165 requested a full tablet report, sending...
W20240503 14:01:18.590273 4191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:18.590788 3657 ts_manager.cc:194] Registered new tserver with Master: 6d05d524f254418bb351065633a4606c (127.0.99.4:42471)
W20240503 14:01:18.591459 4193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:18.591522 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39666
I20240503 14:01:18.591746 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:18.591800 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:18.591822 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744878591822 us; error 0 us; skew 500 ppm
I20240503 14:01:18.592984 396 webserver.cc:460] Webserver started at http://127.0.99.5:42175/ using document root <none> and password file <none>
I20240503 14:01:18.593147 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:18.593192 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:18.593262 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:18.593632 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-4-root/instance:
uuid: "e4f6fd3611bf4f0c812a03bde535fb67"
format_stamp: "Formatted at 2024-05-03 14:01:18 on dist-test-slave-6w8v"
I20240503 14:01:18.595074 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:18.596189 4198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.596458 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:18.596534 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-4-root
uuid: "e4f6fd3611bf4f0c812a03bde535fb67"
format_stamp: "Formatted at 2024-05-03 14:01:18 on dist-test-slave-6w8v"
I20240503 14:01:18.596621 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:18.614894 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:18.615383 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:18.615844 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:18.616359 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:18.616400 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.616442 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:18.616469 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.624068 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.5:37127
I20240503 14:01:18.624161 4261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.5:37127 every 8 connection(s)
I20240503 14:01:18.625262 396 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:01:18.627956 4265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:01:18.628585 4268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:18.629067 396 server_base.cc:1034] running on GCE node
I20240503 14:01:18.629628 4262 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35165
I20240503 14:01:18.629722 4262 heartbeater.cc:464] Registering TS with master...
W20240503 14:01:18.629873 4266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:01:18.629979 4262 heartbeater.cc:511] Master 127.0.99.62:35165 requested a full tablet report, sending...
I20240503 14:01:18.630126 396 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240503 14:01:18.630172 396 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240503 14:01:18.630188 396 hybrid_clock.cc:648] HybridClock initialized: now 1714744878630187 us; error 0 us; skew 500 ppm
I20240503 14:01:18.630620 3657 ts_manager.cc:194] Registered new tserver with Master: e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127)
I20240503 14:01:18.631188 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39682
I20240503 14:01:18.631282 396 webserver.cc:460] Webserver started at http://127.0.99.6:36287/ using document root <none> and password file <none>
I20240503 14:01:18.631417 396 fs_manager.cc:359] Metadata directory not provided
I20240503 14:01:18.631467 396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:01:18.631547 396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:01:18.631938 396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-5-root/instance:
uuid: "06740b673b674aecadcbb0fa2bcac68e"
format_stamp: "Formatted at 2024-05-03 14:01:18 on dist-test-slave-6w8v"
I20240503 14:01:18.633384 396 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240503 14:01:18.634239 4273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.634491 396 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:18.634558 396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-5-root
uuid: "06740b673b674aecadcbb0fa2bcac68e"
format_stamp: "Formatted at 2024-05-03 14:01:18 on dist-test-slave-6w8v"
I20240503 14:01:18.634629 396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:01:18.651899 396 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:01:18.652374 396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:01:18.652717 396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:01:18.653199 396 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:01:18.653240 396 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.653283 396 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:01:18.653312 396 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:01:18.660279 396 rpc_server.cc:307] RPC server started. Bound to: 127.0.99.6:33057
I20240503 14:01:18.660348 4336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.99.6:33057 every 8 connection(s)
I20240503 14:01:18.665050 4337 heartbeater.cc:347] Connected to a master server at 127.0.99.62:35165
I20240503 14:01:18.665158 4337 heartbeater.cc:464] Registering TS with master...
I20240503 14:01:18.665369 4337 heartbeater.cc:511] Master 127.0.99.62:35165 requested a full tablet report, sending...
I20240503 14:01:18.665912 3657 ts_manager.cc:194] Registered new tserver with Master: 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057)
I20240503 14:01:18.666430 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39692
I20240503 14:01:19.356413 3760 consensus_queue.cc:237] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 19, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:19.357977 3836 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:19.358268 3914 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:19.358640 4342 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:19.358878 4113 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:19.360633 4011 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:19.362852 3836 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:19.362852 3914 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:19.364569 3657 catalog_manager.cc:5484] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index -1 to 28, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:01:19.366598 3729 consensus_peers.cc:485] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71dd0142015e4fb791413d533b9ccb55. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:19.369415 3836 consensus_queue.cc:237] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:19.371006 3914 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 14:01:19.371006 3760 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 14:01:19.371410 4109 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:01:19.371783 4349 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:01:19.373046 4109 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
W20240503 14:01:19.375314 3804 consensus_peers.cc:485] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b7539be4b0c41d1accce62f9abbc9e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:19.375972 3914 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:19.375993 3760 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:19.376124 3660 catalog_manager.cc:5484] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index -1 to 26, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:19.376236 3836 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:19.378568 3760 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:19.378599 3914 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:19.379276 4109 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:19.379503 4354 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:19.382572 4109 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
W20240503 14:01:19.383800 3806 consensus_peers.cc:485] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 -> Peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471): Couldn't send request to peer 6d05d524f254418bb351065633a4606c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1bde81d4ddf4de3a2e16be46d97b1b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:19.384002 3760 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:19.385113 3659 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:19.386413 3914 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:19.391494 3914 consensus_queue.cc:237] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 1.24, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:19.393249 3836 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 14:01:19.393291 3760 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 24. Preceding OpId from leader: term: 1 index: 25. (index mismatch)
I20240503 14:01:19.393657 4356 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 14:01:19.393944 4357 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20240503 14:01:19.395514 4357 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:19.396150 3760 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:19.396186 3836 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Committing config change with OpId 1.25: config changed from index -1 to 25, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:19.398792 3657 catalog_manager.cc:5484] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: config changed from index -1 to 25, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:19.398954 3760 consensus_queue.cc:237] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
W20240503 14:01:19.400455 3881 consensus_peers.cc:485] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 -> Peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471): Couldn't send request to peer 6d05d524f254418bb351065633a4606c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11c78a8dc746409aa487dcd7ab58cc82. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:19.401757 3836 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:19.402228 4342 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:19.402345 3914 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:19.402748 4342 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240503 14:01:19.404011 3728 consensus_peers.cc:485] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d121433c26cb4d89876ce434ba669006. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:19.404889 4113 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:19.405336 3914 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:19.405346 3836 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:19.407682 3659 catalog_manager.cc:5484] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index -1 to 27, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:19.408058 3914 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:19.409260 3760 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:19.409647 4003 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:19.409852 3836 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:19.410398 4003 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:19.411322 4357 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:19.411768 3760 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:19.414126 3660 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index -1 to 27, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:19.414399 3836 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
W20240503 14:01:19.418224 3880 consensus_peers.cc:485] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0171f05f97a24f97ab8732925b360c82. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:19.592533 4186 heartbeater.cc:503] Master 127.0.99.62:35165 was elected leader, sending a full tablet report...
I20240503 14:01:19.632076 4262 heartbeater.cc:503] Master 127.0.99.62:35165 was elected leader, sending a full tablet report...
I20240503 14:01:19.667181 4337 heartbeater.cc:503] Master 127.0.99.62:35165 was elected leader, sending a full tablet report...
I20240503 14:01:19.767069 4360 ts_tablet_manager.cc:902] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: Initiating tablet copy from peer 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:19.767683 4360 tablet_copy_client.cc:308] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:40655
I20240503 14:01:19.771363 3850 tablet_copy_service.cc:140] P 2d8cd9c1e8e2469888742770004dcb11: Received BeginTabletCopySession request for tablet 6b7539be4b0c41d1accce62f9abbc9e9 from peer 06740b673b674aecadcbb0fa2bcac68e ({username='slave'} at 127.0.0.1:36498)
I20240503 14:01:19.771472 3850 tablet_copy_service.cc:161] P 2d8cd9c1e8e2469888742770004dcb11: Beginning new tablet copy session on tablet 6b7539be4b0c41d1accce62f9abbc9e9 from peer 06740b673b674aecadcbb0fa2bcac68e at {username='slave'} at 127.0.0.1:36498: session id = 06740b673b674aecadcbb0fa2bcac68e-6b7539be4b0c41d1accce62f9abbc9e9
I20240503 14:01:19.772640 3850 tablet_copy_source_session.cc:205] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:19.773413 4360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b7539be4b0c41d1accce62f9abbc9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:19.775461 4360 tablet_copy_client.cc:783] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:19.775655 4360 tablet_copy_client.cc:647] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:19.776660 4364 tablet_copy_client.cc:1002] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:19.782109 4366 ts_tablet_manager.cc:902] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: Initiating tablet copy from peer f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:19.782727 4366 tablet_copy_client.cc:308] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:43425
I20240503 14:01:19.786818 3925 tablet_copy_service.cc:140] P f6d3c1f17e6b4f42a813bef57d94fe87: Received BeginTabletCopySession request for tablet 11c78a8dc746409aa487dcd7ab58cc82 from peer 6d05d524f254418bb351065633a4606c ({username='slave'} at 127.0.0.1:47998)
I20240503 14:01:19.786926 3925 tablet_copy_service.cc:161] P f6d3c1f17e6b4f42a813bef57d94fe87: Beginning new tablet copy session on tablet 11c78a8dc746409aa487dcd7ab58cc82 from peer 6d05d524f254418bb351065633a4606c at {username='slave'} at 127.0.0.1:47998: session id = 6d05d524f254418bb351065633a4606c-11c78a8dc746409aa487dcd7ab58cc82
I20240503 14:01:19.788079 3925 tablet_copy_source_session.cc:205] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:19.788887 4366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c78a8dc746409aa487dcd7ab58cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:19.791273 4366 tablet_copy_client.cc:783] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:19.791462 4366 tablet_copy_client.cc:647] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:19.792586 4370 tablet_copy_client.cc:1002] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:19.837821 4372 ts_tablet_manager.cc:902] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: Initiating tablet copy from peer f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053)
I20240503 14:01:19.838428 4372 tablet_copy_client.cc:308] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:38053
I20240503 14:01:19.842065 3774 tablet_copy_service.cc:140] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Received BeginTabletCopySession request for tablet d121433c26cb4d89876ce434ba669006 from peer 06740b673b674aecadcbb0fa2bcac68e ({username='slave'} at 127.0.0.1:52368)
I20240503 14:01:19.842180 3774 tablet_copy_service.cc:161] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Beginning new tablet copy session on tablet d121433c26cb4d89876ce434ba669006 from peer 06740b673b674aecadcbb0fa2bcac68e at {username='slave'} at 127.0.0.1:52368: session id = 06740b673b674aecadcbb0fa2bcac68e-d121433c26cb4d89876ce434ba669006
I20240503 14:01:19.843190 3774 tablet_copy_source_session.cc:205] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:19.844139 4372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d121433c26cb4d89876ce434ba669006. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:19.846240 4372 tablet_copy_client.cc:783] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:19.846405 4372 tablet_copy_client.cc:647] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:19.847661 4376 tablet_copy_client.cc:1002] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:19.895972 4377 ts_tablet_manager.cc:902] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: Initiating tablet copy from peer 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:19.896602 4377 tablet_copy_client.cc:308] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:40655
I20240503 14:01:19.900110 3850 tablet_copy_service.cc:140] P 2d8cd9c1e8e2469888742770004dcb11: Received BeginTabletCopySession request for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 from peer 6d05d524f254418bb351065633a4606c ({username='slave'} at 127.0.0.1:36502)
I20240503 14:01:19.900226 3850 tablet_copy_service.cc:161] P 2d8cd9c1e8e2469888742770004dcb11: Beginning new tablet copy session on tablet b1bde81d4ddf4de3a2e16be46d97b1b5 from peer 6d05d524f254418bb351065633a4606c at {username='slave'} at 127.0.0.1:36502: session id = 6d05d524f254418bb351065633a4606c-b1bde81d4ddf4de3a2e16be46d97b1b5
I20240503 14:01:19.901280 3850 tablet_copy_source_session.cc:205] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:19.902078 4377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bde81d4ddf4de3a2e16be46d97b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:19.904639 4377 tablet_copy_client.cc:783] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:19.904825 4377 tablet_copy_client.cc:647] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:19.905740 4381 tablet_copy_client.cc:1002] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:19.916426 4382 ts_tablet_manager.cc:902] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: Initiating tablet copy from peer f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053)
I20240503 14:01:19.916939 4382 tablet_copy_client.cc:308] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:38053
I20240503 14:01:19.920640 3774 tablet_copy_service.cc:140] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Received BeginTabletCopySession request for tablet 71dd0142015e4fb791413d533b9ccb55 from peer e4f6fd3611bf4f0c812a03bde535fb67 ({username='slave'} at 127.0.0.1:52370)
I20240503 14:01:19.920751 3774 tablet_copy_service.cc:161] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Beginning new tablet copy session on tablet 71dd0142015e4fb791413d533b9ccb55 from peer e4f6fd3611bf4f0c812a03bde535fb67 at {username='slave'} at 127.0.0.1:52370: session id = e4f6fd3611bf4f0c812a03bde535fb67-71dd0142015e4fb791413d533b9ccb55
I20240503 14:01:19.921627 3774 tablet_copy_source_session.cc:205] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:19.922370 4382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71dd0142015e4fb791413d533b9ccb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:19.924398 4382 tablet_copy_client.cc:783] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:19.924582 4382 tablet_copy_client.cc:647] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:19.925472 4386 tablet_copy_client.cc:1002] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:20.032637 4388 ts_tablet_manager.cc:902] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: Initiating tablet copy from peer f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:20.033430 4388 tablet_copy_client.cc:308] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:43425
I20240503 14:01:20.037212 3925 tablet_copy_service.cc:140] P f6d3c1f17e6b4f42a813bef57d94fe87: Received BeginTabletCopySession request for tablet 0171f05f97a24f97ab8732925b360c82 from peer e4f6fd3611bf4f0c812a03bde535fb67 ({username='slave'} at 127.0.0.1:48012)
I20240503 14:01:20.037370 3925 tablet_copy_service.cc:161] P f6d3c1f17e6b4f42a813bef57d94fe87: Beginning new tablet copy session on tablet 0171f05f97a24f97ab8732925b360c82 from peer e4f6fd3611bf4f0c812a03bde535fb67 at {username='slave'} at 127.0.0.1:48012: session id = e4f6fd3611bf4f0c812a03bde535fb67-0171f05f97a24f97ab8732925b360c82
I20240503 14:01:20.038422 3925 tablet_copy_source_session.cc:205] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:20.039209 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0171f05f97a24f97ab8732925b360c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:20.041638 4388 tablet_copy_client.cc:783] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:20.042045 4388 tablet_copy_client.cc:647] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:20.043619 4392 tablet_copy_client.cc:1002] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:20.435469 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:20.437443 3914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
mode: GRACEFUL
new_leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:47982
I20240503 14:01:20.437633 3914 raft_consensus.cc:595] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Received request to transfer leadership to TS f71f8fdeb4004bb8b8012dc8b79ff8e1
I20240503 14:01:20.442327 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
new_leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:52372
I20240503 14:01:20.442495 3762 raft_consensus.cc:595] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership to TS f6d3c1f17e6b4f42a813bef57d94fe87
I20240503 14:01:20.442834 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 14:01:20.443029 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:20.618717 4396 raft_consensus.cc:979] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: : Instructing follower f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:20.618839 4342 raft_consensus.cc:1067] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Signalling peer f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:20.619213 3914 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:47962
I20240503 14:01:20.619352 3914 raft_consensus.cc:482] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:20.619408 3914 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:20.620291 3914 raft_consensus.cc:504] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:20.620644 3914 leader_election.cc:288] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 3 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:20.620966 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:20.621083 3840 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:20.622246 3840 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 3.
I20240503 14:01:20.622524 3880 leader_election.cc:302] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:20.622876 4400 raft_consensus.cc:2781] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:01:20.622973 4400 raft_consensus.cc:686] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:20.622984 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:20.623092 3760 raft_consensus.cc:3032] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:01:20.623136 3760 raft_consensus.cc:729] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:20.623111 4400 consensus_queue.cc:237] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:20.623265 3760 consensus_queue.cc:260] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:20.623427 3760 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:20.624473 3760 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 3.
I20240503 14:01:20.625221 3659 catalog_manager.cc:5484] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 2 to 3, leader changed from f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } }
I20240503 14:01:20.672922 4393 raft_consensus.cc:979] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: : Instructing follower f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:20.673064 4400 raft_consensus.cc:1067] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Signalling peer f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:20.673509 3760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:52334
I20240503 14:01:20.673636 3760 raft_consensus.cc:482] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:20.673700 3760 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:20.674814 3760 raft_consensus.cc:504] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:20.675171 3760 leader_election.cc:288] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:20.675494 3836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:20.675493 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:20.675690 3836 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:20.675753 3914 raft_consensus.cc:3032] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:20.675807 3914 raft_consensus.cc:729] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:20.675951 3914 consensus_queue.cc:260] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:20.676113 3914 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:20.676688 3836 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:20.676836 3914 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:20.676964 3729 leader_election.cc:302] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:20.677155 4342 raft_consensus.cc:2781] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:20.677263 4342 raft_consensus.cc:686] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:20.677400 4342 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:20.679114 3659 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 1 to 2, leader changed from f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:20.685456 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:20.685617 3760 raft_consensus.cc:595] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:01:20.690390 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:20.690534 3760 raft_consensus.cc:595] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership
W20240503 14:01:20.690770 3648 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 0171f05f97a24f97ab8732925b360c82 already in progress [suppressed 9 similar messages]
I20240503 14:01:20.691407 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:20.691509 3760 raft_consensus.cc:595] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership
W20240503 14:01:20.691812 3709 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 0171f05f97a24f97ab8732925b360c82 already in progress
W20240503 14:01:20.691871 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 0171f05f97a24f97ab8732925b360c82 already in progress
I20240503 14:01:21.014658 3836 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 14:01:21.015056 4400 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:21.017956 3762 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20240503 14:01:21.018702 4400 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240503 14:01:21.215332 3729 consensus_peers.cc:485] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:21.290334 3914 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:21.290896 4396 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:21.292678 4342 raft_consensus.cc:979] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: : Instructing follower f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:21.292801 4397 raft_consensus.cc:1067] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Signalling peer f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:21.293236 3913 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:47962
I20240503 14:01:21.293396 3913 raft_consensus.cc:482] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:21.293455 3913 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:21.294457 3913 raft_consensus.cc:504] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:21.294865 3913 leader_election.cc:288] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 3 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:21.295327 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:21.295449 3760 raft_consensus.cc:3032] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:01:21.295493 3760 raft_consensus.cc:729] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:21.295600 3836 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:21.295667 3760 consensus_queue.cc:260] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:21.295938 3760 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:21.296902 3760 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 3.
I20240503 14:01:21.297307 3881 leader_election.cc:302] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:21.297387 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:21.297574 3837 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:21.297623 4400 raft_consensus.cc:2781] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:01:21.297703 4400 raft_consensus.cc:686] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:21.297839 4400 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:21.298806 3837 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 3.
I20240503 14:01:21.301447 3657 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 2 to 3, leader changed from f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:21.777209 4360 tablet_copy_client.cc:523] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:21.778815 4360 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap starting.
I20240503 14:01:21.792997 4366 tablet_copy_client.cc:523] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:21.794454 4366 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: Bootstrap starting.
I20240503 14:01:21.796422 3762 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:21.796891 4417 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240503 14:01:21.799428 3880 consensus_peers.cc:485] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:21.800060 3840 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:21.800441 4400 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:21.800634 4360 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=99 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:21.800895 4360 tablet_bootstrap.cc:492] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap complete.
I20240503 14:01:21.801127 4360 ts_tablet_manager.cc:1367] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.002s
I20240503 14:01:21.801774 4360 raft_consensus.cc:348] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:21.801993 4360 raft_consensus.cc:729] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06740b673b674aecadcbb0fa2bcac68e, State: Initialized, Role: LEARNER
I20240503 14:01:21.802104 4360 consensus_queue.cc:260] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:21.802500 4360 ts_tablet_manager.cc:1398] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:21.803588 3850 tablet_copy_service.cc:342] P 2d8cd9c1e8e2469888742770004dcb11: Request end of tablet copy session 06740b673b674aecadcbb0fa2bcac68e-6b7539be4b0c41d1accce62f9abbc9e9 received from {username='slave'} at 127.0.0.1:36498
I20240503 14:01:21.803671 3850 tablet_copy_service.cc:433] P 2d8cd9c1e8e2469888742770004dcb11: ending tablet copy session 06740b673b674aecadcbb0fa2bcac68e-6b7539be4b0c41d1accce62f9abbc9e9 on tablet 6b7539be4b0c41d1accce62f9abbc9e9 with peer 06740b673b674aecadcbb0fa2bcac68e
I20240503 14:01:21.821239 4366 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: Bootstrap replayed 1/1 log segments. Stats: ops{read=25 overwritten=0 applied=25 ignored=0} inserts{seen=91 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:21.821532 4366 tablet_bootstrap.cc:492] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: Bootstrap complete.
I20240503 14:01:21.821709 4366 ts_tablet_manager.cc:1367] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.008s
I20240503 14:01:21.822261 4366 raft_consensus.cc:348] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:21.822404 4366 raft_consensus.cc:729] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d05d524f254418bb351065633a4606c, State: Initialized, Role: LEARNER
I20240503 14:01:21.822511 4366 consensus_queue.cc:260] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 25 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:21.822927 4366 ts_tablet_manager.cc:1398] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240503 14:01:21.823359 3925 tablet_copy_service.cc:342] P f6d3c1f17e6b4f42a813bef57d94fe87: Request end of tablet copy session 6d05d524f254418bb351065633a4606c-11c78a8dc746409aa487dcd7ab58cc82 received from {username='slave'} at 127.0.0.1:47998
I20240503 14:01:21.823426 3925 tablet_copy_service.cc:433] P f6d3c1f17e6b4f42a813bef57d94fe87: ending tablet copy session 6d05d524f254418bb351065633a4606c-11c78a8dc746409aa487dcd7ab58cc82 on tablet 11c78a8dc746409aa487dcd7ab58cc82 with peer 6d05d524f254418bb351065633a4606c
I20240503 14:01:21.848090 4372 tablet_copy_client.cc:523] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:21.850455 4372 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap starting.
I20240503 14:01:21.874962 4372 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:21.875319 4372 tablet_bootstrap.cc:492] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap complete.
I20240503 14:01:21.875582 4372 ts_tablet_manager.cc:1367] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240503 14:01:21.876142 4372 raft_consensus.cc:348] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:21.876268 4372 raft_consensus.cc:729] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06740b673b674aecadcbb0fa2bcac68e, State: Initialized, Role: LEARNER
I20240503 14:01:21.876372 4372 consensus_queue.cc:260] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:21.876688 4372 ts_tablet_manager.cc:1398] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240503 14:01:21.877256 3774 tablet_copy_service.cc:342] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Request end of tablet copy session 06740b673b674aecadcbb0fa2bcac68e-d121433c26cb4d89876ce434ba669006 received from {username='slave'} at 127.0.0.1:52368
I20240503 14:01:21.877359 3774 tablet_copy_service.cc:433] P f71f8fdeb4004bb8b8012dc8b79ff8e1: ending tablet copy session 06740b673b674aecadcbb0fa2bcac68e-d121433c26cb4d89876ce434ba669006 on tablet d121433c26cb4d89876ce434ba669006 with peer 06740b673b674aecadcbb0fa2bcac68e
I20240503 14:01:21.906150 4377 tablet_copy_client.cc:523] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:21.908041 4377 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: Bootstrap starting.
I20240503 14:01:21.926049 4382 tablet_copy_client.cc:523] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:21.927801 4382 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap starting.
I20240503 14:01:21.930263 4377 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:21.930521 4377 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: Bootstrap complete.
I20240503 14:01:21.930686 4377 ts_tablet_manager.cc:1367] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.004s
I20240503 14:01:21.931104 4377 raft_consensus.cc:348] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:21.931221 4377 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d05d524f254418bb351065633a4606c, State: Initialized, Role: LEARNER
I20240503 14:01:21.931309 4377 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:21.931624 4377 ts_tablet_manager.cc:1398] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240503 14:01:21.932256 3850 tablet_copy_service.cc:342] P 2d8cd9c1e8e2469888742770004dcb11: Request end of tablet copy session 6d05d524f254418bb351065633a4606c-b1bde81d4ddf4de3a2e16be46d97b1b5 received from {username='slave'} at 127.0.0.1:36502
I20240503 14:01:21.932332 3850 tablet_copy_service.cc:433] P 2d8cd9c1e8e2469888742770004dcb11: ending tablet copy session 6d05d524f254418bb351065633a4606c-b1bde81d4ddf4de3a2e16be46d97b1b5 on tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with peer 6d05d524f254418bb351065633a4606c
I20240503 14:01:21.950816 4382 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:21.951151 4382 tablet_bootstrap.cc:492] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap complete.
I20240503 14:01:21.951361 4382 ts_tablet_manager.cc:1367] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.004s
I20240503 14:01:21.951954 4382 raft_consensus.cc:348] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:21.952093 4382 raft_consensus.cc:729] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e4f6fd3611bf4f0c812a03bde535fb67, State: Initialized, Role: LEARNER
I20240503 14:01:21.952201 4382 consensus_queue.cc:260] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:21.952561 4382 ts_tablet_manager.cc:1398] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67: Time spent starting tablet: real 0.001s user 0.002s sys 0.003s
I20240503 14:01:21.952982 3774 tablet_copy_service.cc:342] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Request end of tablet copy session e4f6fd3611bf4f0c812a03bde535fb67-71dd0142015e4fb791413d533b9ccb55 received from {username='slave'} at 127.0.0.1:52370
I20240503 14:01:21.953068 3774 tablet_copy_service.cc:433] P f71f8fdeb4004bb8b8012dc8b79ff8e1: ending tablet copy session e4f6fd3611bf4f0c812a03bde535fb67-71dd0142015e4fb791413d533b9ccb55 on tablet 71dd0142015e4fb791413d533b9ccb55 with peer e4f6fd3611bf4f0c812a03bde535fb67
I20240503 14:01:21.974540 4160 raft_consensus.cc:1203] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 14:01:22.028749 4311 raft_consensus.cc:1203] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20240503 14:01:22.044088 4388 tablet_copy_client.cc:523] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:22.045768 4388 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap starting.
I20240503 14:01:22.053659 4311 raft_consensus.cc:1203] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 14:01:22.069089 4388 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:22.069396 4388 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap complete.
I20240503 14:01:22.069590 4388 ts_tablet_manager.cc:1367] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: Time spent bootstrapping tablet: real 0.024s user 0.022s sys 0.001s
I20240503 14:01:22.070139 4388 raft_consensus.cc:348] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:22.070268 4388 raft_consensus.cc:729] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4f6fd3611bf4f0c812a03bde535fb67, State: Initialized, Role: LEARNER
I20240503 14:01:22.070374 4388 consensus_queue.cc:260] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:22.070675 4388 ts_tablet_manager.cc:1398] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:22.071092 3925 tablet_copy_service.cc:342] P f6d3c1f17e6b4f42a813bef57d94fe87: Request end of tablet copy session e4f6fd3611bf4f0c812a03bde535fb67-0171f05f97a24f97ab8732925b360c82 received from {username='slave'} at 127.0.0.1:48012
I20240503 14:01:22.071183 3925 tablet_copy_service.cc:433] P f6d3c1f17e6b4f42a813bef57d94fe87: ending tablet copy session e4f6fd3611bf4f0c812a03bde535fb67-0171f05f97a24f97ab8732925b360c82 on tablet 0171f05f97a24f97ab8732925b360c82 with peer e4f6fd3611bf4f0c812a03bde535fb67
I20240503 14:01:22.229928 4236 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 1 LEARNER]: Advancing to term 3
I20240503 14:01:22.230163 4160 raft_consensus.cc:1203] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.24->[1.25-1.25] Dedup: 1.25->[]
I20240503 14:01:22.231266 4236 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 LEARNER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:22.231671 4400 consensus_queue.cc:1037] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:22.233538 4400 raft_consensus.cc:1050] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: attempting to promote NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 to VOTER
I20240503 14:01:22.233937 4400 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:22.235231 3840 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:22.235656 4236 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 LEARNER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:22.235999 3762 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:22.235962 4417 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:22.236157 4435 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:22.236367 4393 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:22.237808 4436 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 27 to 30, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.238379 3762 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.238584 4236 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.240731 3659 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index 27 to 30, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.241282 3840 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.246569 3913 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:22.247761 4236 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:22.247754 3840 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:22.248157 4435 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:22.248354 4436 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:22.250396 4236 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.251040 3840 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.249848 4435 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.253149 3657 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 reported cstate change: config changed from index 30 to 31, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New cstate: current_term: 3 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.254163 3647 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0171f05f97a24f97ab8732925b360c82 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:22.255548 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet 0171f05f97a24f97ab8732925b360c82 with delete_type TABLET_DATA_TOMBSTONED (TS f71f8fdeb4004bb8b8012dc8b79ff8e1 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:22.256142 4441 tablet_replica.cc:330] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: stopping tablet replica
I20240503 14:01:22.256284 4441 raft_consensus.cc:2229] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:22.256475 4441 raft_consensus.cc:2258] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:22.257081 4441 ts_tablet_manager.cc:1875] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:22.259538 4441 ts_tablet_manager.cc:1888] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20240503 14:01:22.259663 4441 log.cc:1196] T 0171f05f97a24f97ab8732925b360c82 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root/wals/0171f05f97a24f97ab8732925b360c82
I20240503 14:01:22.260339 3648 catalog_manager.cc:4830] TS f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053): tablet 0171f05f97a24f97ab8732925b360c82 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:22.434561 4236 raft_consensus.cc:1203] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240503 14:01:22.443257 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:22.445279 3914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
mode: GRACEFUL
new_leader_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:47982
I20240503 14:01:22.445468 3914 raft_consensus.cc:595] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Received request to transfer leadership to TS 2d8cd9c1e8e2469888742770004dcb11
I20240503 14:01:22.446187 3836 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:22.446317 3836 raft_consensus.cc:595] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Received request to transfer leadership to TS f71f8fdeb4004bb8b8012dc8b79ff8e1
I20240503 14:01:22.446969 3914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
mode: GRACEFUL
new_leader_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:47982
I20240503 14:01:22.447098 3914 raft_consensus.cc:595] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Received request to transfer leadership to TS 2d8cd9c1e8e2469888742770004dcb11
I20240503 14:01:22.447360 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 14:01:22.447506 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:22.452154 4396 raft_consensus.cc:1050] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1: attempting to promote NON_VOTER 06740b673b674aecadcbb0fa2bcac68e to VOTER
I20240503 14:01:22.452643 4396 consensus_queue.cc:237] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:22.454020 4311 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:01:22.454139 3914 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:01:22.454412 4445 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:22.454509 3836 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:01:22.454663 4396 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:22.454833 4444 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:22.456501 4445 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.456735 3836 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.458314 4311 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.458606 3914 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.460532 3656 catalog_manager.cc:5484] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 27 to 28, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.464295 3762 consensus_queue.cc:237] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:22.465458 4311 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:01:22.465494 3836 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240503 14:01:22.465797 4444 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:22.465996 4396 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:22.467464 4442 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.468178 4311 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.470395 3657 catalog_manager.cc:5484] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index 28 to 29, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:22.471465 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d121433c26cb4d89876ce434ba669006 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:22.471891 3836 raft_consensus.cc:2932] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.472841 3904 tablet_service.cc:1514] Processing DeleteTablet for tablet d121433c26cb4d89876ce434ba669006 with delete_type TABLET_DATA_TOMBSTONED (TS f6d3c1f17e6b4f42a813bef57d94fe87 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:47958
I20240503 14:01:22.473448 4450 tablet_replica.cc:330] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: stopping tablet replica
I20240503 14:01:22.473605 4450 raft_consensus.cc:2229] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:22.475198 4450 raft_consensus.cc:2258] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:22.475884 4450 ts_tablet_manager.cc:1875] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:22.478541 4450 ts_tablet_manager.cc:1888] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20240503 14:01:22.478659 4450 log.cc:1196] T d121433c26cb4d89876ce434ba669006 P f6d3c1f17e6b4f42a813bef57d94fe87: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root/wals/d121433c26cb4d89876ce434ba669006
I20240503 14:01:22.479142 3647 catalog_manager.cc:4830] TS f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425): tablet d121433c26cb4d89876ce434ba669006 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:22.508188 4394 raft_consensus.cc:1050] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11: attempting to promote NON_VOTER 06740b673b674aecadcbb0fa2bcac68e to VOTER
I20240503 14:01:22.508679 4394 consensus_queue.cc:237] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:22.509969 4311 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:22.509991 3914 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:22.510241 3762 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:22.510454 4440 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:22.510692 4452 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:22.510954 4394 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:22.512364 4440 raft_consensus.cc:979] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:22.512465 4440 raft_consensus.cc:1067] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Signalling peer f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:22.512850 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:52346
I20240503 14:01:22.512964 3762 raft_consensus.cc:482] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:22.513022 3762 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:22.513141 4440 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.514185 3762 raft_consensus.cc:504] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:22.514542 3762 leader_election.cc:288] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:22.514968 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:22.515072 3914 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:22.515899 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:22.515995 3840 raft_consensus.cc:3032] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:22.516027 3840 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:22.516098 3914 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:22.515973 4311 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.516024 3660 catalog_manager.cc:5484] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 26 to 27, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:22.516198 3840 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:22.516396 3840 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:22.516474 3729 leader_election.cc:302] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:22.516978 3914 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.517196 3840 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:22.517299 4445 raft_consensus.cc:2781] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:22.517372 4445 raft_consensus.cc:686] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:22.517501 4445 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:22.517717 3762 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.519702 3656 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 1 to 2, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:22.520679 3836 consensus_queue.cc:237] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:22.521754 4311 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:01:22.522055 4452 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:22.522478 3760 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240503 14:01:22.523027 4421 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:22.524088 4452 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.524449 4311 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.525457 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:22.525568 3760 raft_consensus.cc:595] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:01:22.525729 3647 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6b7539be4b0c41d1accce62f9abbc9e9 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:22.526060 3760 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.526664 3659 catalog_manager.cc:5484] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e reported cstate change: config changed from index 27 to 28, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:22.530408 3904 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b7539be4b0c41d1accce62f9abbc9e9 with delete_type TABLET_DATA_TOMBSTONED (TS f6d3c1f17e6b4f42a813bef57d94fe87 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:47958
I20240503 14:01:22.530436 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:22.530546 3762 raft_consensus.cc:595] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:01:22.530763 4450 tablet_replica.cc:330] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: stopping tablet replica
I20240503 14:01:22.530900 4450 raft_consensus.cc:2229] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:22.531057 4450 raft_consensus.cc:2258] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:22.531397 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:22.531553 3762 raft_consensus.cc:595] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:01:22.531697 4450 ts_tablet_manager.cc:1875] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:01:22.531890 3709 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for b1bde81d4ddf4de3a2e16be46d97b1b5 already in progress
W20240503 14:01:22.531972 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for b1bde81d4ddf4de3a2e16be46d97b1b5 already in progress
I20240503 14:01:22.534122 4450 ts_tablet_manager.cc:1888] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20240503 14:01:22.534199 4450 log.cc:1196] T 6b7539be4b0c41d1accce62f9abbc9e9 P f6d3c1f17e6b4f42a813bef57d94fe87: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root/wals/6b7539be4b0c41d1accce62f9abbc9e9
I20240503 14:01:22.534689 3647 catalog_manager.cc:4830] TS f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425): tablet 6b7539be4b0c41d1accce62f9abbc9e9 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:22.810002 4436 raft_consensus.cc:1050] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87: attempting to promote NON_VOTER 6d05d524f254418bb351065633a4606c to VOTER
I20240503 14:01:22.810434 4436 consensus_queue.cc:237] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:22.811739 4160 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 14:01:22.811807 3836 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 14:01:22.811933 3760 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20240503 14:01:22.812070 4436 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:01:22.812507 4436 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:01:22.812666 4460 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:01:22.814563 4436 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:22.815239 3760 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:22.814958 4160 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:22.817711 3656 catalog_manager.cc:5484] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index 25 to 26, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:22.818557 3836 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index 25 to 26, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
W20240503 14:01:22.821460 4466 consensus_queue.cc:1159] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2d8cd9c1e8e2469888742770004dcb11" responder_term: 1 status { last_received { term: 1 index: 26 } last_committed_idx: 26 last_received_current_leader { term: 1 index: 26 } }
I20240503 14:01:22.821667 3914 consensus_queue.cc:237] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:22.822602 3760 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:22.822680 4160 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:22.822970 4461 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:22.823133 4466 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:22.824816 4436 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:22.826462 4160 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:22.826473 3760 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:22.828058 3647 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11c78a8dc746409aa487dcd7ab58cc82 with cas_config_opid_index 26: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:22.828575 3660 catalog_manager.cc:5484] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index 26 to 27, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:22.831219 3827 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c78a8dc746409aa487dcd7ab58cc82 with delete_type TABLET_DATA_TOMBSTONED (TS 2d8cd9c1e8e2469888742770004dcb11 not found in new config with opid_index 27) from {username='slave'} at 127.0.0.1:36456
I20240503 14:01:22.831646 4467 tablet_replica.cc:330] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: stopping tablet replica
I20240503 14:01:22.831934 4467 raft_consensus.cc:2229] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:22.832130 4467 raft_consensus.cc:2258] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:22.832686 4467 ts_tablet_manager.cc:1875] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:22.834898 4467 ts_tablet_manager.cc:1888] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.26
I20240503 14:01:22.834986 4467 log.cc:1196] T 11c78a8dc746409aa487dcd7ab58cc82 P 2d8cd9c1e8e2469888742770004dcb11: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root/wals/11c78a8dc746409aa487dcd7ab58cc82
I20240503 14:01:22.835466 3647 catalog_manager.cc:4830] TS 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655): tablet 11c78a8dc746409aa487dcd7ab58cc82 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:22.891495 4436 raft_consensus.cc:979] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87: : Instructing follower 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:22.891665 4436 raft_consensus.cc:1067] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Signalling peer 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:22.892047 3836 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:36462
I20240503 14:01:22.892212 3836 raft_consensus.cc:482] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:22.892269 3836 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:22.893400 3836 raft_consensus.cc:504] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:22.893777 3836 leader_election.cc:288] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 2 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:22.894168 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:22.894248 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:22.894299 3760 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:22.894352 3914 raft_consensus.cc:3032] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:22.894388 3914 raft_consensus.cc:729] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:22.894517 3914 consensus_queue.cc:260] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:22.894673 3914 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:22.895375 3760 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 2.
I20240503 14:01:22.895687 3806 leader_election.cc:302] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:22.895706 3914 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 2.
I20240503 14:01:22.895898 4440 raft_consensus.cc:2781] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:22.895988 4440 raft_consensus.cc:686] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:22.896147 4440 consensus_queue.cc:237] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:22.897868 3656 catalog_manager.cc:5484] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 1 to 2, leader changed from f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:22.922425 4466 raft_consensus.cc:979] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87: : Instructing follower 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:22.922590 4466 raft_consensus.cc:1067] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Signalling peer 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:22.922966 3836 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:36462
I20240503 14:01:22.923090 3836 raft_consensus.cc:482] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:22.923158 3836 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:22.924249 3836 raft_consensus.cc:504] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:22.924649 3836 leader_election.cc:288] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 4 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:22.924875 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:22.924933 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:22.925061 3760 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:22.925061 3914 raft_consensus.cc:3032] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:01:22.925169 3914 raft_consensus.cc:729] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:22.925307 3914 consensus_queue.cc:260] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:22.925470 3914 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:22.926177 3760 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 4.
I20240503 14:01:22.926319 3914 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 4.
I20240503 14:01:22.926440 3806 leader_election.cc:302] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:22.926610 4440 raft_consensus.cc:2781] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:01:22.926682 4440 raft_consensus.cc:686] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:22.926806 4440 consensus_queue.cc:237] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:22.928395 3656 catalog_manager.cc:5484] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 3 to 4, leader changed from f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 4 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:22.966418 4445 raft_consensus.cc:1050] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1: attempting to promote NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 to VOTER
I20240503 14:01:22.966890 4445 consensus_queue.cc:237] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 19, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:22.968500 4236 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 LEARNER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:22.968627 3836 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:22.968505 3914 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:22.969095 4396 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:22.969273 4444 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:22.969580 4449 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:22.971853 4396 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.974493 3836 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.974828 3656 catalog_manager.cc:5484] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index 28 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:22.974890 3914 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.977201 4236 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.980166 3762 consensus_queue.cc:237] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 19, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:22.980839 4160 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Advancing to term 2
I20240503 14:01:22.981262 3914 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:22.981262 4236 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:22.981694 4449 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:22.981879 4444 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:22.982060 4160 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 2 LEARNER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:22.982729 4444 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:22.983551 4444 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.983878 3914 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.984275 4236 raft_consensus.cc:2932] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.985934 3659 catalog_manager.cc:5484] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: config changed from index 29 to 30, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:22.986665 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71dd0142015e4fb791413d533b9ccb55 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:22.986891 4396 raft_consensus.cc:1050] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: attempting to promote NON_VOTER 6d05d524f254418bb351065633a4606c to VOTER
W20240503 14:01:22.986968 4396 raft_consensus.cc:1053] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Unable to promote non-voter 6d05d524f254418bb351065633a4606c: Service unavailable: leader transfer in progress
I20240503 14:01:22.988540 3827 tablet_service.cc:1514] Processing DeleteTablet for tablet 71dd0142015e4fb791413d533b9ccb55 with delete_type TABLET_DATA_TOMBSTONED (TS 2d8cd9c1e8e2469888742770004dcb11 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:36456
I20240503 14:01:22.988770 4467 tablet_replica.cc:330] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: stopping tablet replica
I20240503 14:01:22.988907 4467 raft_consensus.cc:2229] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:22.989039 4467 raft_consensus.cc:2258] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:22.989815 4467 ts_tablet_manager.cc:1875] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:22.991658 4467 ts_tablet_manager.cc:1888] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 14:01:22.991735 4467 log.cc:1196] T 71dd0142015e4fb791413d533b9ccb55 P 2d8cd9c1e8e2469888742770004dcb11: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root/wals/71dd0142015e4fb791413d533b9ccb55
I20240503 14:01:22.992205 3647 catalog_manager.cc:4830] TS 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655): tablet 71dd0142015e4fb791413d533b9ccb55 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:23.081058 3836 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:23.081524 4444 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:23.083715 4449 raft_consensus.cc:979] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: : Instructing follower 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:23.083818 4449 raft_consensus.cc:1067] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Signalling peer 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:23.084273 3840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:36472
I20240503 14:01:23.084499 3840 raft_consensus.cc:482] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:23.084569 3840 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:23.085343 3914 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:23.085757 4449 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:23.085776 3840 raft_consensus.cc:504] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:23.086112 4449 raft_consensus.cc:1050] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: attempting to promote NON_VOTER 6d05d524f254418bb351065633a4606c to VOTER
W20240503 14:01:23.086184 4449 raft_consensus.cc:1053] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Unable to promote non-voter 6d05d524f254418bb351065633a4606c: Service unavailable: leader transfer in progress
I20240503 14:01:23.086282 3840 leader_election.cc:288] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 3 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:23.087044 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:23.087172 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:23.087263 3760 raft_consensus.cc:3032] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:01:23.087306 3760 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:23.087430 3760 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:23.087666 3760 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:23.088862 3760 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 3.
I20240503 14:01:23.089170 3806 leader_election.cc:302] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters: f6d3c1f17e6b4f42a813bef57d94fe87
I20240503 14:01:23.089381 4421 raft_consensus.cc:2781] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:01:23.089475 4421 raft_consensus.cc:686] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:23.089628 4421 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:23.091500 3660 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 2 to 3, leader changed from f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 3 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:23.356889 3760 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:01:23.357298 4421 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:23.359858 3914 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:01:23.360383 4421 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:23.383147 3914 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 14:01:23.383627 4421 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:23.386096 3760 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20240503 14:01:23.386452 4487 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:23.541065 4160 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 2 LEARNER]: Advancing to term 3
I20240503 14:01:23.541121 3760 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:23.541568 4487 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:23.542189 4160 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 LEARNER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:23.542629 4487 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:23.544559 4487 raft_consensus.cc:1050] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: attempting to promote NON_VOTER 6d05d524f254418bb351065633a4606c to VOTER
I20240503 14:01:23.544906 4487 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:23.545472 3914 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:23.546675 3914 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:23.546960 3913 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:23.547274 4160 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 LEARNER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:23.547377 3762 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:23.547418 4489 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:23.547927 4489 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:23.548182 4489 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:23.549463 4440 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.550200 3913 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.551254 3760 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.551307 4160 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.552788 3660 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: config changed from index 27 to 30, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.556259 3840 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:23.557355 4160 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:23.557454 3913 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:23.557745 4487 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:23.557958 4421 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:23.559777 4421 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.560418 4160 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.561784 3647 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:23.562796 3659 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c reported cstate change: config changed from index 30 to 31, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New cstate: current_term: 3 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.563853 3913 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:23.565877 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with delete_type TABLET_DATA_TOMBSTONED (TS f71f8fdeb4004bb8b8012dc8b79ff8e1 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:23.566228 4490 tablet_replica.cc:330] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: stopping tablet replica
I20240503 14:01:23.566370 4490 raft_consensus.cc:2229] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:23.566551 4490 raft_consensus.cc:2258] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:23.567292 4490 ts_tablet_manager.cc:1875] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:23.569813 4490 ts_tablet_manager.cc:1888] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20240503 14:01:23.569906 4490 log.cc:1196] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root/wals/b1bde81d4ddf4de3a2e16be46d97b1b5
I20240503 14:01:23.570411 3648 catalog_manager.cc:4830] TS f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053): tablet b1bde81d4ddf4de3a2e16be46d97b1b5 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:23.992717 3913 consensus_queue.cc:237] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:23.994076 3760 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:23.994076 3840 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240503 14:01:23.994271 3881 consensus_peers.cc:485] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 -> Peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471): Couldn't send request to peer 6d05d524f254418bb351065633a4606c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95a2d80baec84cefb6d8b5e0ca27e5d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:23.994436 4436 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:23.994603 4492 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:23.996336 4466 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:23.996783 3840 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:23.996783 3760 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:23.999082 3656 catalog_manager.cc:5484] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New cstate: current_term: 1 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:23.999764 3840 consensus_queue.cc:237] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:24.001240 3760 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:24.001741 3913 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:24.002225 4489 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:24.003899 4489 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:24.004585 4440 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:24.006134 3913 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
W20240503 14:01:24.006429 3805 consensus_peers.cc:485] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95d4c1bc93314dbea0dc3a714c8c9ec6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:24.007215 3760 consensus_queue.cc:237] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:24.007980 3659 catalog_manager.cc:5484] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index -1 to 27, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New cstate: current_term: 1 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:24.009006 3840 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20240503 14:01:24.009168 3728 consensus_peers.cc:485] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11893d5b97174d7389005a373091703b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:24.009227 3913 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240503 14:01:24.009537 4444 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:24.009553 3760 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:24.009866 4444 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:24.011796 4444 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:24.012257 3913 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:24.014142 3840 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:24.014218 3913 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:24.015498 4236 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 14:01:24.015784 3836 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 14:01:24.015815 3657 catalog_manager.cc:5484] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: config changed from index -1 to 27, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New cstate: current_term: 1 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:24.016253 4466 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 14:01:24.016556 4466 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 14:01:24.018095 4436 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:24.018460 4236 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:24.020452 3836 consensus_queue.cc:237] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:24.021193 3840 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:24.021673 3913 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:24.022029 4421 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240503 14:01:24.022670 3879 consensus_peers.cc:485] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0171f05f97a24f97ab8732925b360c82. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:24.023094 3760 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:24.023420 4440 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240503 14:01:24.023888 3806 consensus_peers.cc:485] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 -> Peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471): Couldn't send request to peer 6d05d524f254418bb351065633a4606c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d6ad905b31541ffa6f2007d886b7643. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:24.023944 3659 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 reported cstate change: config changed from index 31 to 32, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New cstate: current_term: 3 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:24.024369 4440 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:24.026158 3760 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:24.026964 3913 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:24.027340 3657 catalog_manager.cc:5484] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index -1 to 28, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New cstate: current_term: 2 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:24.028370 3840 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:24.029639 4160 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
W20240503 14:01:24.029652 3805 consensus_peers.cc:485] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1bde81d4ddf4de3a2e16be46d97b1b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:24.029623 3913 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 14:01:24.030010 4487 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 14:01:24.030202 4421 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 14:01:24.031590 4440 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:24.032311 3913 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:24.032390 4160 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:24.034369 3659 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 31 to 32, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New cstate: current_term: 3 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:24.399108 4512 ts_tablet_manager.cc:902] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: Initiating tablet copy from peer f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:24.399865 4512 tablet_copy_client.cc:308] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:43425
I20240503 14:01:24.400274 3925 tablet_copy_service.cc:140] P f6d3c1f17e6b4f42a813bef57d94fe87: Received BeginTabletCopySession request for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 from peer 6d05d524f254418bb351065633a4606c ({username='slave'} at 127.0.0.1:47998)
I20240503 14:01:24.400382 3925 tablet_copy_service.cc:161] P f6d3c1f17e6b4f42a813bef57d94fe87: Beginning new tablet copy session on tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 from peer 6d05d524f254418bb351065633a4606c at {username='slave'} at 127.0.0.1:47998: session id = 6d05d524f254418bb351065633a4606c-95a2d80baec84cefb6d8b5e0ca27e5d7
I20240503 14:01:24.401412 3925 tablet_copy_source_session.cc:205] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:24.402278 4512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:24.404870 4512 tablet_copy_client.cc:783] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:24.405068 4512 tablet_copy_client.cc:647] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:24.406137 4516 tablet_copy_client.cc:1002] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:24.408852 4517 ts_tablet_manager.cc:902] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: Initiating tablet copy from peer 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:24.409478 4517 tablet_copy_client.cc:308] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:40655
I20240503 14:01:24.413353 3850 tablet_copy_service.cc:140] P 2d8cd9c1e8e2469888742770004dcb11: Received BeginTabletCopySession request for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 from peer e4f6fd3611bf4f0c812a03bde535fb67 ({username='slave'} at 127.0.0.1:44094)
I20240503 14:01:24.413494 3850 tablet_copy_service.cc:161] P 2d8cd9c1e8e2469888742770004dcb11: Beginning new tablet copy session on tablet b1bde81d4ddf4de3a2e16be46d97b1b5 from peer e4f6fd3611bf4f0c812a03bde535fb67 at {username='slave'} at 127.0.0.1:44094: session id = e4f6fd3611bf4f0c812a03bde535fb67-b1bde81d4ddf4de3a2e16be46d97b1b5
I20240503 14:01:24.414510 3850 tablet_copy_source_session.cc:205] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:24.415274 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bde81d4ddf4de3a2e16be46d97b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:24.417677 4517 tablet_copy_client.cc:783] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:24.417868 4517 tablet_copy_client.cc:647] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:24.419178 4521 tablet_copy_client.cc:1002] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:24.447795 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:24.449551 3913 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
mode: GRACEFUL
new_leader_uuid: "e4f6fd3611bf4f0c812a03bde535fb67"
from {username='slave'} at 127.0.0.1:47982
I20240503 14:01:24.449721 3913 raft_consensus.cc:595] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Received request to transfer leadership to TS e4f6fd3611bf4f0c812a03bde535fb67
I20240503 14:01:24.450453 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11893d5b97174d7389005a373091703b"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
new_leader_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:52372
I20240503 14:01:24.450567 3760 raft_consensus.cc:595] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Received request to transfer leadership to TS 2d8cd9c1e8e2469888742770004dcb11
I20240503 14:01:24.451256 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6b7539be4b0c41d1accce62f9abbc9e9"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "06740b673b674aecadcbb0fa2bcac68e"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:24.451382 3840 raft_consensus.cc:595] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Received request to transfer leadership to TS 06740b673b674aecadcbb0fa2bcac68e
I20240503 14:01:24.452044 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:24.452152 3840 raft_consensus.cc:595] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Received request to transfer leadership to TS f71f8fdeb4004bb8b8012dc8b79ff8e1
I20240503 14:01:24.452807 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "6d05d524f254418bb351065633a4606c"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:24.452908 3840 raft_consensus.cc:595] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Received request to transfer leadership to TS 6d05d524f254418bb351065633a4606c
I20240503 14:01:24.453514 3836 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:24.453631 3836 raft_consensus.cc:595] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Received request to transfer leadership to TS f6d3c1f17e6b4f42a813bef57d94fe87
I20240503 14:01:24.453881 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240503 14:01:24.454013 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:24.470080 4524 ts_tablet_manager.cc:902] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: Initiating tablet copy from peer f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:24.470755 4524 tablet_copy_client.cc:308] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.3:43425
I20240503 14:01:24.474725 3925 tablet_copy_service.cc:140] P f6d3c1f17e6b4f42a813bef57d94fe87: Received BeginTabletCopySession request for tablet 0171f05f97a24f97ab8732925b360c82 from peer 06740b673b674aecadcbb0fa2bcac68e ({username='slave'} at 127.0.0.1:53252)
I20240503 14:01:24.474893 3925 tablet_copy_service.cc:161] P f6d3c1f17e6b4f42a813bef57d94fe87: Beginning new tablet copy session on tablet 0171f05f97a24f97ab8732925b360c82 from peer 06740b673b674aecadcbb0fa2bcac68e at {username='slave'} at 127.0.0.1:53252: session id = 06740b673b674aecadcbb0fa2bcac68e-0171f05f97a24f97ab8732925b360c82
I20240503 14:01:24.476007 3925 tablet_copy_source_session.cc:205] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:24.476776 4524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0171f05f97a24f97ab8732925b360c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:24.479260 4524 tablet_copy_client.cc:783] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:24.479445 4524 tablet_copy_client.cc:647] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:24.480502 4528 tablet_copy_client.cc:1002] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:24.505807 4531 ts_tablet_manager.cc:902] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: Initiating tablet copy from peer 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:24.506538 4531 tablet_copy_client.cc:308] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:40655
I20240503 14:01:24.506904 3850 tablet_copy_service.cc:140] P 2d8cd9c1e8e2469888742770004dcb11: Received BeginTabletCopySession request for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 from peer e4f6fd3611bf4f0c812a03bde535fb67 ({username='slave'} at 127.0.0.1:44094)
I20240503 14:01:24.507009 3850 tablet_copy_service.cc:161] P 2d8cd9c1e8e2469888742770004dcb11: Beginning new tablet copy session on tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 from peer e4f6fd3611bf4f0c812a03bde535fb67 at {username='slave'} at 127.0.0.1:44094: session id = e4f6fd3611bf4f0c812a03bde535fb67-95d4c1bc93314dbea0dc3a714c8c9ec6
I20240503 14:01:24.508093 3850 tablet_copy_source_session.cc:205] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:24.508852 4531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:24.511157 4531 tablet_copy_client.cc:783] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:24.511358 4531 tablet_copy_client.cc:647] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:24.512763 4535 tablet_copy_client.cc:1002] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:24.513152 4536 ts_tablet_manager.cc:902] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: Initiating tablet copy from peer 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:24.513794 4536 tablet_copy_client.cc:308] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.2:40655
I20240503 14:01:24.514211 3850 tablet_copy_service.cc:140] P 2d8cd9c1e8e2469888742770004dcb11: Received BeginTabletCopySession request for tablet 8d6ad905b31541ffa6f2007d886b7643 from peer 6d05d524f254418bb351065633a4606c ({username='slave'} at 127.0.0.1:36502)
I20240503 14:01:24.514297 3850 tablet_copy_service.cc:161] P 2d8cd9c1e8e2469888742770004dcb11: Beginning new tablet copy session on tablet 8d6ad905b31541ffa6f2007d886b7643 from peer 6d05d524f254418bb351065633a4606c at {username='slave'} at 127.0.0.1:36502: session id = 6d05d524f254418bb351065633a4606c-8d6ad905b31541ffa6f2007d886b7643
I20240503 14:01:24.515164 3850 tablet_copy_source_session.cc:205] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:24.515949 4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6ad905b31541ffa6f2007d886b7643. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:24.518200 4536 tablet_copy_client.cc:783] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:24.518366 4536 tablet_copy_client.cc:647] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:24.519542 4540 tablet_copy_client.cc:1002] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:24.533244 4541 ts_tablet_manager.cc:902] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: Initiating tablet copy from peer f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053)
I20240503 14:01:24.533784 4541 tablet_copy_client.cc:308] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:38053
I20240503 14:01:24.534186 3774 tablet_copy_service.cc:140] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Received BeginTabletCopySession request for tablet 11893d5b97174d7389005a373091703b from peer 06740b673b674aecadcbb0fa2bcac68e ({username='slave'} at 127.0.0.1:52368)
I20240503 14:01:24.534288 3774 tablet_copy_service.cc:161] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Beginning new tablet copy session on tablet 11893d5b97174d7389005a373091703b from peer 06740b673b674aecadcbb0fa2bcac68e at {username='slave'} at 127.0.0.1:52368: session id = 06740b673b674aecadcbb0fa2bcac68e-11893d5b97174d7389005a373091703b
I20240503 14:01:24.535162 3774 tablet_copy_source_session.cc:205] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:24.536098 4541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11893d5b97174d7389005a373091703b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:24.538733 4541 tablet_copy_client.cc:783] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:24.538930 4541 tablet_copy_client.cc:647] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:24.540001 4543 tablet_copy_client.cc:1002] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:24.612319 4505 raft_consensus.cc:979] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: : Instructing follower 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:24.612462 4505 raft_consensus.cc:1067] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Signalling peer 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:24.612820 3836 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11893d5b97174d7389005a373091703b"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:36472
I20240503 14:01:24.612973 3836 raft_consensus.cc:482] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:24.613041 3836 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:24.613919 3836 raft_consensus.cc:504] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:24.614287 3836 leader_election.cc:288] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 2 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:24.614639 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11893d5b97174d7389005a373091703b" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:24.614639 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11893d5b97174d7389005a373091703b" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:24.614790 3913 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:24.614789 3760 raft_consensus.cc:3032] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:24.614851 3760 raft_consensus.cc:729] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:24.614991 3760 consensus_queue.cc:260] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:24.615190 3760 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:24.616117 3913 raft_consensus.cc:2454] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 2.
I20240503 14:01:24.616194 3760 raft_consensus.cc:2454] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 2.
I20240503 14:01:24.616411 3805 leader_election.cc:302] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:24.616596 4489 raft_consensus.cc:2781] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:24.616773 4489 raft_consensus.cc:686] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:24.616946 4489 consensus_queue.cc:237] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:24.618525 3657 catalog_manager.cc:5484] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 1 to 2, leader changed from f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:24.622675 4547 raft_consensus.cc:979] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower 6d05d524f254418bb351065633a4606c to start an election
I20240503 14:01:24.622776 4547 raft_consensus.cc:1067] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Signalling peer 6d05d524f254418bb351065633a4606c to start an election
I20240503 14:01:24.623157 4160 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5"
dest_uuid: "6d05d524f254418bb351065633a4606c"
from {username='slave'} at 127.0.0.1:48018
I20240503 14:01:24.623298 4160 raft_consensus.cc:482] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:24.623355 4160 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:24.624157 4436 raft_consensus.cc:979] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87: : Instructing follower e4f6fd3611bf4f0c812a03bde535fb67 to start an election
I20240503 14:01:24.624245 4436 raft_consensus.cc:1067] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Signalling peer e4f6fd3611bf4f0c812a03bde535fb67 to start an election
I20240503 14:01:24.624614 4236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82"
dest_uuid: "e4f6fd3611bf4f0c812a03bde535fb67"
from {username='slave'} at 127.0.0.1:49602
I20240503 14:01:24.624728 4236 raft_consensus.cc:482] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:24.624697 4160 raft_consensus.cc:504] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:24.624780 4236 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:24.625169 4160 leader_election.cc:288] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [CANDIDATE]: Term 4 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:24.625407 3836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "6d05d524f254418bb351065633a4606c" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:24.625517 3836 raft_consensus.cc:3032] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:01:24.625581 3836 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:24.625494 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1bde81d4ddf4de3a2e16be46d97b1b5" candidate_uuid: "6d05d524f254418bb351065633a4606c" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:24.625707 3913 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:24.625748 3836 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:24.625963 3836 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:24.625794 4236 raft_consensus.cc:504] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:24.626437 4236 leader_election.cc:288] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [CANDIDATE]: Term 4 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:24.626631 3913 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d05d524f254418bb351065633a4606c in term 4.
I20240503 14:01:24.626927 4125 leader_election.cc:302] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d05d524f254418bb351065633a4606c, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:24.626986 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:24.627074 3913 raft_consensus.cc:3032] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:01:24.627111 3913 raft_consensus.cc:729] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:24.626937 3836 raft_consensus.cc:2454] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d05d524f254418bb351065633a4606c in term 4.
I20240503 14:01:24.627264 3913 consensus_queue.cc:260] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:24.626940 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0171f05f97a24f97ab8732925b360c82" candidate_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:24.627498 3840 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:24.627528 3913 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:24.627763 4552 raft_consensus.cc:2781] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:01:24.627979 4552 raft_consensus.cc:686] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 4 LEADER]: Becoming Leader. State: Replica: 6d05d524f254418bb351065633a4606c, State: Running, Role: LEADER
I20240503 14:01:24.628167 4552 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:24.628429 3840 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f6fd3611bf4f0c812a03bde535fb67 in term 4.
I20240503 14:01:24.628429 3913 raft_consensus.cc:2454] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f6fd3611bf4f0c812a03bde535fb67 in term 4.
I20240503 14:01:24.628825 4201 leader_election.cc:302] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4f6fd3611bf4f0c812a03bde535fb67, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:24.629094 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11893d5b97174d7389005a373091703b"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36456
I20240503 14:01:24.629209 3840 raft_consensus.cc:595] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:01:24.629433 4554 raft_consensus.cc:2781] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:01:24.629675 4554 raft_consensus.cc:686] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 LEADER]: Becoming Leader. State: Replica: e4f6fd3611bf4f0c812a03bde535fb67, State: Running, Role: LEADER
I20240503 14:01:24.630080 4554 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:24.630648 3657 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c reported cstate change: term changed from 3 to 4, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to 6d05d524f254418bb351065633a4606c (127.0.99.4). New cstate: current_term: 4 leader_uuid: "6d05d524f254418bb351065633a4606c" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:24.631989 3656 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 reported cstate change: term changed from 3 to 4, leader changed from f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) to e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5). New cstate: current_term: 4 leader_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:24.635233 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11893d5b97174d7389005a373091703b"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36456
I20240503 14:01:24.635381 3840 raft_consensus.cc:595] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Received request to transfer leadership
I20240503 14:01:24.636525 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11893d5b97174d7389005a373091703b"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36456
I20240503 14:01:24.636689 3840 raft_consensus.cc:595] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Received request to transfer leadership
W20240503 14:01:24.637030 3709 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 11893d5b97174d7389005a373091703b already in progress
W20240503 14:01:24.637082 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 11893d5b97174d7389005a373091703b already in progress
I20240503 14:01:24.637305 4489 raft_consensus.cc:979] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:24.637400 4547 raft_consensus.cc:1067] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Signalling peer f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:24.637809 3760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:52346
I20240503 14:01:24.637943 3760 raft_consensus.cc:482] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:24.638012 3760 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:24.639276 3760 raft_consensus.cc:504] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:24.639683 3760 leader_election.cc:288] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 3 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:24.640143 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:24.640144 3913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:24.640295 3840 raft_consensus.cc:3032] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:01:24.640300 3913 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:24.640348 3840 raft_consensus.cc:729] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:24.640496 3840 consensus_queue.cc:260] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:24.640697 3840 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:24.641299 3913 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 3.
I20240503 14:01:24.641654 3840 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 3.
I20240503 14:01:24.641912 3729 leader_election.cc:302] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:24.642194 4505 raft_consensus.cc:2781] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:01:24.642282 4505 raft_consensus.cc:686] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:24.642431 4505 consensus_queue.cc:237] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:24.644443 3656 catalog_manager.cc:5484] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 2 to 3, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:24.649065 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:24.649211 3760 raft_consensus.cc:595] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Received request to transfer leadership
I20240503 14:01:24.657218 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:24.657392 3760 raft_consensus.cc:595] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Received request to transfer leadership
I20240503 14:01:24.658105 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:24.658227 3760 raft_consensus.cc:595] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Received request to transfer leadership
W20240503 14:01:24.658535 3709 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 8d6ad905b31541ffa6f2007d886b7643 already in progress
W20240503 14:01:24.658596 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 8d6ad905b31541ffa6f2007d886b7643 already in progress
I20240503 14:01:24.847419 4489 raft_consensus.cc:979] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:24.847551 4489 raft_consensus.cc:1067] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Signalling peer f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:24.847951 3913 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:47972
I20240503 14:01:24.848088 3913 raft_consensus.cc:482] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:24.848147 3913 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:01:24.849218 3913 raft_consensus.cc:504] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:24.849592 3913 leader_election.cc:288] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 5 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:24.849882 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:24.850000 3760 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:01:24.850035 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc6eba6dbb2c4efda6d24f9ee9d6e8f7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:24.850108 3840 raft_consensus.cc:3032] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:01:24.850140 3840 raft_consensus.cc:729] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:24.850283 3840 consensus_queue.cc:260] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:24.850447 3840 raft_consensus.cc:3037] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:01:24.851120 3760 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 5.
I20240503 14:01:24.851315 3840 raft_consensus.cc:2454] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 5.
I20240503 14:01:24.851378 3881 leader_election.cc:302] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:24.851781 4551 raft_consensus.cc:2781] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:01:24.851884 4551 raft_consensus.cc:686] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 5 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:24.852020 4551 consensus_queue.cc:237] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:24.853654 3656 catalog_manager.cc:5484] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 4 to 5, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 5 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } }
I20240503 14:01:24.866612 4547 raft_consensus.cc:979] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower 06740b673b674aecadcbb0fa2bcac68e to start an election
I20240503 14:01:24.866746 4489 raft_consensus.cc:1067] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Signalling peer 06740b673b674aecadcbb0fa2bcac68e to start an election
I20240503 14:01:24.867123 4311 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6b7539be4b0c41d1accce62f9abbc9e9"
dest_uuid: "06740b673b674aecadcbb0fa2bcac68e"
from {username='slave'} at 127.0.0.1:38364
I20240503 14:01:24.867260 4311 raft_consensus.cc:482] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:24.867327 4311 raft_consensus.cc:3037] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:24.868520 4311 raft_consensus.cc:504] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:24.868999 4311 leader_election.cc:288] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [CANDIDATE]: Term 2 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655)
I20240503 14:01:24.869287 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7539be4b0c41d1accce62f9abbc9e9" candidate_uuid: "06740b673b674aecadcbb0fa2bcac68e" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:24.869426 3837 raft_consensus.cc:3032] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:24.869472 3837 raft_consensus.cc:729] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:24.869607 3837 consensus_queue.cc:260] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:24.869766 3837 raft_consensus.cc:3037] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:24.870510 3837 raft_consensus.cc:2454] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06740b673b674aecadcbb0fa2bcac68e in term 2.
I20240503 14:01:24.869287 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b7539be4b0c41d1accce62f9abbc9e9" candidate_uuid: "06740b673b674aecadcbb0fa2bcac68e" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:24.870807 3760 raft_consensus.cc:3037] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:24.870810 4276 leader_election.cc:302] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06740b673b674aecadcbb0fa2bcac68e, 2d8cd9c1e8e2469888742770004dcb11; no voters:
I20240503 14:01:24.871260 4561 raft_consensus.cc:2781] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:24.871469 4561 raft_consensus.cc:686] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 2 LEADER]: Becoming Leader. State: Replica: 06740b673b674aecadcbb0fa2bcac68e, State: Running, Role: LEADER
I20240503 14:01:24.871686 3760 raft_consensus.cc:2454] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06740b673b674aecadcbb0fa2bcac68e in term 2.
I20240503 14:01:24.871639 4561 consensus_queue.cc:237] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:24.873077 3656 catalog_manager.cc:5484] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e reported cstate change: term changed from 1 to 2, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6). New cstate: current_term: 2 leader_uuid: "06740b673b674aecadcbb0fa2bcac68e" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:01:25.020594 4200 consensus_peers.cc:485] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:25.022605 3760 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:25.023034 4489 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:25.024096 3913 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 14:01:25.024441 4552 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 14:01:25.026468 4489 raft_consensus.cc:979] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:25.026559 4489 raft_consensus.cc:1067] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Signalling peer f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:25.027079 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11893d5b97174d7389005a373091703b"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:52346
W20240503 14:01:25.027647 3804 consensus_peers.cc:485] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:25.027812 3762 raft_consensus.cc:482] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:25.027896 3762 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:25.029067 3762 raft_consensus.cc:504] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:25.029453 3762 leader_election.cc:288] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 3 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:25.029585 3837 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 14:01:25.029789 3914 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:25.030231 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11893d5b97174d7389005a373091703b" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:25.030329 3914 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:25.030474 4571 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:25.031047 4552 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 14:01:25.031491 3914 raft_consensus.cc:2454] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 3.
I20240503 14:01:25.031667 3913 raft_consensus.cc:1226] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Rejecting Update request from peer 2d8cd9c1e8e2469888742770004dcb11 for earlier term 2. Current term is 3. Ops: [2.28-2.28]
I20240503 14:01:25.031952 3729 leader_election.cc:302] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:25.032240 4546 raft_consensus.cc:2781] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:01:25.032346 4546 raft_consensus.cc:686] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:25.032614 4546 consensus_queue.cc:237] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:25.032289 4571 consensus_queue.cc:1046] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: INVALID_TERM, Last received: 1.27, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:25.032902 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11893d5b97174d7389005a373091703b" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:25.033259 3837 raft_consensus.cc:3032] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:01:25.033322 3837 raft_consensus.cc:729] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:25.033479 3837 consensus_queue.cc:260] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:25.033664 3837 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:25.034857 3837 raft_consensus.cc:2454] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 3.
W20240503 14:01:25.035246 4125 consensus_peers.cc:485] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:25.037330 3656 catalog_manager.cc:5484] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 2 to 3, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 3 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:25.063357 3837 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:25.063793 4546 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:25.065627 4546 raft_consensus.cc:979] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: : Instructing follower 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:25.065721 4546 raft_consensus.cc:1067] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Signalling peer 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:25.066061 3840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:36472
I20240503 14:01:25.066179 3840 raft_consensus.cc:482] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:25.066229 3840 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:25.066619 3913 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:25.066931 4546 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:25.067487 3840 raft_consensus.cc:504] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:25.067924 3840 leader_election.cc:288] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 4 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:25.068202 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:25.069232 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:25.069339 3762 raft_consensus.cc:3032] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Stepping down as leader of term 3
I20240503 14:01:25.069378 3762 raft_consensus.cc:729] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
W20240503 14:01:25.069433 3730 consensus_peers.cc:485] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 -> Peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471): Couldn't send request to peer 6d05d524f254418bb351065633a4606c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:25.069525 3762 consensus_queue.cc:260] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:25.069701 3762 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:25.070817 3762 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 4.
I20240503 14:01:25.071329 3806 leader_election.cc:302] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters: f6d3c1f17e6b4f42a813bef57d94fe87
I20240503 14:01:25.071506 4571 raft_consensus.cc:2781] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Leader election won for term 4
I20240503 14:01:25.071621 4571 raft_consensus.cc:686] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:25.071748 4571 consensus_queue.cc:237] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:25.073390 3657 catalog_manager.cc:5484] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 3 to 4, leader changed from f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 4 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:25.075395 3840 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 14:01:25.075809 4554 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 14:01:25.078536 3913 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20240503 14:01:25.078923 4554 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 14:01:25.318020 3840 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 5 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240503 14:01:25.318516 4564 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:25.321063 3762 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 5 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20240503 14:01:25.321576 4436 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:25.343281 3840 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer 06740b673b674aecadcbb0fa2bcac68e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:25.344379 4561 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:25.347329 3762 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 06740b673b674aecadcbb0fa2bcac68e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:25.347776 4562 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:25.497982 3914 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Advancing to term 4
I20240503 14:01:25.499357 3914 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 14:01:25.499770 4571 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:25.501932 3762 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20240503 14:01:25.502341 4571 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240503 14:01:25.502600 3806 consensus_peers.cc:485] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 -> Peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471): Couldn't send request to peer 6d05d524f254418bb351065633a4606c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:25.583657 3837 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:25.584185 4546 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240503 14:01:25.586584 3728 consensus_peers.cc:485] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:25.586908 3914 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:25.587692 4546 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:26.406448 4512 tablet_copy_client.cc:523] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:26.407945 4512 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: Bootstrap starting.
I20240503 14:01:26.419473 4517 tablet_copy_client.cc:523] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:26.421172 4517 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap starting.
I20240503 14:01:26.430485 4512 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=102 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:26.430815 4512 tablet_bootstrap.cc:492] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: Bootstrap complete.
I20240503 14:01:26.431031 4512 ts_tablet_manager.cc:1367] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.002s
I20240503 14:01:26.431950 4512 raft_consensus.cc:348] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:26.432113 4512 raft_consensus.cc:729] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6d05d524f254418bb351065633a4606c, State: Initialized, Role: LEARNER
I20240503 14:01:26.432211 4512 consensus_queue.cc:260] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:26.432777 4512 ts_tablet_manager.cc:1398] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:01:26.433235 3925 tablet_copy_service.cc:342] P f6d3c1f17e6b4f42a813bef57d94fe87: Request end of tablet copy session 6d05d524f254418bb351065633a4606c-95a2d80baec84cefb6d8b5e0ca27e5d7 received from {username='slave'} at 127.0.0.1:47998
I20240503 14:01:26.433326 3925 tablet_copy_service.cc:433] P f6d3c1f17e6b4f42a813bef57d94fe87: ending tablet copy session 6d05d524f254418bb351065633a4606c-95a2d80baec84cefb6d8b5e0ca27e5d7 on tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 with peer 6d05d524f254418bb351065633a4606c
I20240503 14:01:26.446772 4517 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:26.447109 4517 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap complete.
I20240503 14:01:26.447357 4517 ts_tablet_manager.cc:1367] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: Time spent bootstrapping tablet: real 0.026s user 0.021s sys 0.001s
I20240503 14:01:26.447901 4517 raft_consensus.cc:348] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:26.448050 4517 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: e4f6fd3611bf4f0c812a03bde535fb67, State: Initialized, Role: LEARNER
I20240503 14:01:26.448148 4517 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:26.448474 4517 ts_tablet_manager.cc:1398] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:01:26.448943 3850 tablet_copy_service.cc:342] P 2d8cd9c1e8e2469888742770004dcb11: Request end of tablet copy session e4f6fd3611bf4f0c812a03bde535fb67-b1bde81d4ddf4de3a2e16be46d97b1b5 received from {username='slave'} at 127.0.0.1:44094
I20240503 14:01:26.449036 3850 tablet_copy_service.cc:433] P 2d8cd9c1e8e2469888742770004dcb11: ending tablet copy session e4f6fd3611bf4f0c812a03bde535fb67-b1bde81d4ddf4de3a2e16be46d97b1b5 on tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with peer e4f6fd3611bf4f0c812a03bde535fb67
I20240503 14:01:26.454324 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:26.456194 3914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
mode: GRACEFUL
new_leader_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:47982
I20240503 14:01:26.456362 3914 raft_consensus.cc:595] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Received request to transfer leadership to TS 2d8cd9c1e8e2469888742770004dcb11
I20240503 14:01:26.457072 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:26.457209 3840 raft_consensus.cc:595] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Received request to transfer leadership to TS f71f8fdeb4004bb8b8012dc8b79ff8e1
I20240503 14:01:26.458025 3762 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d121433c26cb4d89876ce434ba669006"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
new_leader_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:52372
I20240503 14:01:26.458148 3762 raft_consensus.cc:595] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Received request to transfer leadership to TS 2d8cd9c1e8e2469888742770004dcb11
I20240503 14:01:26.458443 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 14:01:26.458575 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:26.480825 4524 tablet_copy_client.cc:523] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:26.482421 4524 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap starting.
I20240503 14:01:26.509265 4524 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:26.509651 4524 tablet_bootstrap.cc:492] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap complete.
I20240503 14:01:26.509792 4564 raft_consensus.cc:979] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87: : Instructing follower 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:26.509899 4564 raft_consensus.cc:1067] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Signalling peer 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:26.509886 4524 ts_tablet_manager.cc:1367] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.007s
I20240503 14:01:26.510324 3837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:36462
I20240503 14:01:26.510454 3837 raft_consensus.cc:482] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:26.510519 3837 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.510648 4524 raft_consensus.cc:348] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:26.510779 4524 raft_consensus.cc:729] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 06740b673b674aecadcbb0fa2bcac68e, State: Initialized, Role: LEARNER
I20240503 14:01:26.510882 4524 consensus_queue.cc:260] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:26.511196 4524 ts_tablet_manager.cc:1398] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:26.511701 3837 raft_consensus.cc:504] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:26.512104 3837 leader_election.cc:288] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 2 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:26.512317 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:26.512426 3762 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.512548 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:26.512651 3914 raft_consensus.cc:3032] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:26.512692 3914 raft_consensus.cc:729] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:26.512851 3914 consensus_queue.cc:260] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:26.513063 4531 tablet_copy_client.cc:523] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:26.513067 3914 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.513254 3925 tablet_copy_service.cc:342] P f6d3c1f17e6b4f42a813bef57d94fe87: Request end of tablet copy session 06740b673b674aecadcbb0fa2bcac68e-0171f05f97a24f97ab8732925b360c82 received from {username='slave'} at 127.0.0.1:53252
I20240503 14:01:26.513319 3925 tablet_copy_service.cc:433] P f6d3c1f17e6b4f42a813bef57d94fe87: ending tablet copy session 06740b673b674aecadcbb0fa2bcac68e-0171f05f97a24f97ab8732925b360c82 on tablet 0171f05f97a24f97ab8732925b360c82 with peer 06740b673b674aecadcbb0fa2bcac68e
I20240503 14:01:26.513553 3762 raft_consensus.cc:2454] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 2.
I20240503 14:01:26.513875 3806 leader_election.cc:302] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:26.514189 3914 raft_consensus.cc:2454] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 2.
I20240503 14:01:26.514393 4571 raft_consensus.cc:2781] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:26.514595 4571 raft_consensus.cc:686] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:26.514765 4531 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap starting.
I20240503 14:01:26.514763 4571 consensus_queue.cc:237] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:26.517782 3657 catalog_manager.cc:5484] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 1 to 2, leader changed from f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:26.519826 4536 tablet_copy_client.cc:523] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:26.521472 4536 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: Bootstrap starting.
I20240503 14:01:26.537886 4531 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:26.538210 4531 tablet_bootstrap.cc:492] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: Bootstrap complete.
I20240503 14:01:26.538408 4531 ts_tablet_manager.cc:1367] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.000s
I20240503 14:01:26.538954 4531 raft_consensus.cc:348] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:26.539115 4531 raft_consensus.cc:729] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4f6fd3611bf4f0c812a03bde535fb67, State: Initialized, Role: LEARNER
I20240503 14:01:26.539224 4531 consensus_queue.cc:260] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:26.539613 4531 ts_tablet_manager.cc:1398] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240503 14:01:26.540045 3850 tablet_copy_service.cc:342] P 2d8cd9c1e8e2469888742770004dcb11: Request end of tablet copy session e4f6fd3611bf4f0c812a03bde535fb67-95d4c1bc93314dbea0dc3a714c8c9ec6 received from {username='slave'} at 127.0.0.1:44094
I20240503 14:01:26.540140 3850 tablet_copy_service.cc:433] P 2d8cd9c1e8e2469888742770004dcb11: ending tablet copy session e4f6fd3611bf4f0c812a03bde535fb67-95d4c1bc93314dbea0dc3a714c8c9ec6 on tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 with peer e4f6fd3611bf4f0c812a03bde535fb67
I20240503 14:01:26.540424 4541 tablet_copy_client.cc:523] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:26.542207 4541 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap starting.
I20240503 14:01:26.556882 4536 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:26.557241 4536 tablet_bootstrap.cc:492] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: Bootstrap complete.
I20240503 14:01:26.557454 4536 ts_tablet_manager.cc:1367] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: Time spent bootstrapping tablet: real 0.036s user 0.023s sys 0.000s
I20240503 14:01:26.558018 4536 raft_consensus.cc:348] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:26.558147 4536 raft_consensus.cc:729] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6d05d524f254418bb351065633a4606c, State: Initialized, Role: LEARNER
I20240503 14:01:26.558243 4536 consensus_queue.cc:260] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:26.558545 4536 ts_tablet_manager.cc:1398] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:01:26.558956 3850 tablet_copy_service.cc:342] P 2d8cd9c1e8e2469888742770004dcb11: Request end of tablet copy session 6d05d524f254418bb351065633a4606c-8d6ad905b31541ffa6f2007d886b7643 received from {username='slave'} at 127.0.0.1:36502
I20240503 14:01:26.559029 3850 tablet_copy_service.cc:433] P 2d8cd9c1e8e2469888742770004dcb11: ending tablet copy session 6d05d524f254418bb351065633a4606c-8d6ad905b31541ffa6f2007d886b7643 on tablet 8d6ad905b31541ffa6f2007d886b7643 with peer 6d05d524f254418bb351065633a4606c
I20240503 14:01:26.565340 4541 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=104 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:26.565651 4541 tablet_bootstrap.cc:492] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap complete.
I20240503 14:01:26.565871 4541 ts_tablet_manager.cc:1367] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.007s
I20240503 14:01:26.566452 4541 raft_consensus.cc:348] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:26.566592 4541 raft_consensus.cc:729] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 06740b673b674aecadcbb0fa2bcac68e, State: Initialized, Role: LEARNER
I20240503 14:01:26.566704 4541 consensus_queue.cc:260] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:26.567003 4541 ts_tablet_manager.cc:1398] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240503 14:01:26.567448 3774 tablet_copy_service.cc:342] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Request end of tablet copy session 06740b673b674aecadcbb0fa2bcac68e-11893d5b97174d7389005a373091703b received from {username='slave'} at 127.0.0.1:52368
I20240503 14:01:26.567525 3774 tablet_copy_service.cc:433] P f71f8fdeb4004bb8b8012dc8b79ff8e1: ending tablet copy session 06740b673b674aecadcbb0fa2bcac68e-11893d5b97174d7389005a373091703b on tablet 11893d5b97174d7389005a373091703b with peer 06740b673b674aecadcbb0fa2bcac68e
I20240503 14:01:26.595649 4236 raft_consensus.cc:1203] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20240503 14:01:26.723498 4236 raft_consensus.cc:3037] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 LEARNER]: Advancing to term 4
I20240503 14:01:26.745242 4578 raft_consensus.cc:979] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1: : Instructing follower 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:26.745363 4578 raft_consensus.cc:1067] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Signalling peer 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:26.745741 3840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d121433c26cb4d89876ce434ba669006"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:36472
I20240503 14:01:26.745869 3840 raft_consensus.cc:482] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:26.745927 3840 raft_consensus.cc:3037] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.747035 3840 raft_consensus.cc:504] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:26.747400 3840 leader_election.cc:288] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 2 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057)
I20240503 14:01:26.747720 3762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d121433c26cb4d89876ce434ba669006" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:26.747792 4311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d121433c26cb4d89876ce434ba669006" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "06740b673b674aecadcbb0fa2bcac68e"
I20240503 14:01:26.747876 3762 raft_consensus.cc:3032] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:26.747915 4311 raft_consensus.cc:3037] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.747920 3762 raft_consensus.cc:729] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:26.748106 3762 consensus_queue.cc:260] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:26.748278 3762 raft_consensus.cc:3037] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.748863 4311 raft_consensus.cc:2454] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 2.
I20240503 14:01:26.749137 3804 leader_election.cc:302] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06740b673b674aecadcbb0fa2bcac68e, 2d8cd9c1e8e2469888742770004dcb11; no voters:
I20240503 14:01:26.749269 3762 raft_consensus.cc:2454] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 2.
I20240503 14:01:26.749366 4571 raft_consensus.cc:2781] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:26.749469 4571 raft_consensus.cc:686] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:26.749609 4571 consensus_queue.cc:237] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:26.750968 4311 raft_consensus.cc:3037] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 1 LEARNER]: Advancing to term 3
I20240503 14:01:26.751283 3657 catalog_manager.cc:5484] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 1 to 2, leader changed from f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 2 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:26.751906 4311 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 3 LEARNER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:26.752580 4578 consensus_queue.cc:1037] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:26.752862 4571 raft_consensus.cc:979] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:26.752930 4571 raft_consensus.cc:1067] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Signalling peer f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:26.753259 3762 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:52346
I20240503 14:01:26.753361 3762 raft_consensus.cc:482] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:26.753412 3762 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.754503 3762 raft_consensus.cc:504] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:26.754868 3762 leader_election.cc:288] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:26.755127 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:26.755177 3837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:26.755234 3914 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.755250 3837 raft_consensus.cc:3032] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:26.755285 3837 raft_consensus.cc:729] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:26.755414 3837 consensus_queue.cc:260] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:26.755633 3837 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:26.756347 3914 raft_consensus.cc:2454] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:26.756510 3837 raft_consensus.cc:2454] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:26.756614 3729 leader_election.cc:302] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:26.756791 4578 raft_consensus.cc:2781] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:26.756981 4578 raft_consensus.cc:686] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:26.757143 4578 consensus_queue.cc:237] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:26.757474 4611 raft_consensus.cc:1050] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: attempting to promote NON_VOTER 06740b673b674aecadcbb0fa2bcac68e to VOTER
I20240503 14:01:26.757870 4611 consensus_queue.cc:237] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:26.759230 3837 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:26.759685 4616 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:26.759872 3914 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:26.760246 4616 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:26.760699 3657 catalog_manager.cc:5484] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 1 to 2, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:26.763142 4311 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 3 LEARNER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240503 14:01:26.763518 4616 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:26.763490 4578 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:26.763885 3914 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:26.764432 4311 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:26.765651 3840 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:26.766171 3657 catalog_manager.cc:5484] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index 27 to 30, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:26.922933 4159 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 2 LEARNER]: Advancing to term 4
I20240503 14:01:26.924284 4159 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 4 LEARNER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240503 14:01:26.924988 4600 consensus_queue.cc:1037] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:26.927284 4571 raft_consensus.cc:1050] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: attempting to promote NON_VOTER 6d05d524f254418bb351065633a4606c to VOTER
I20240503 14:01:26.927762 4571 consensus_queue.cc:237] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 29, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:26.929412 3914 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 14:01:26.929831 4159 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 4 LEARNER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 14:01:26.929836 4600 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:26.930079 3760 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20240503 14:01:26.930616 4600 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:26.930892 4626 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:26.932986 4600 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index 28 to 31, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:26.933971 4159 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 28 to 31, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:26.934239 3762 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 28 to 31, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:26.937438 3660 catalog_manager.cc:5484] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 28 to 31, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:26.938637 3914 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index 28 to 31, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
W20240503 14:01:26.941386 4600 consensus_queue.cc:1159] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" responder_term: 4 status { last_received { term: 4 index: 31 } last_committed_idx: 31 last_received_current_leader { term: 4 index: 31 } }
I20240503 14:01:26.941984 3840 consensus_queue.cc:237] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:26.943631 3914 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 14:01:26.944083 4600 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 14:01:26.945652 4159 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 4 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20240503 14:01:26.946245 4627 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 14:01:26.946277 4626 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:26.947445 4159 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:26.948585 3647 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8d6ad905b31541ffa6f2007d886b7643 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:26.949062 3914 raft_consensus.cc:2932] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:26.949342 3656 catalog_manager.cc:5484] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 31 to 32, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New cstate: current_term: 4 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:26.952818 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6ad905b31541ffa6f2007d886b7643 with delete_type TABLET_DATA_TOMBSTONED (TS f71f8fdeb4004bb8b8012dc8b79ff8e1 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:26.953225 4630 tablet_replica.cc:330] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: stopping tablet replica
I20240503 14:01:26.953393 4630 raft_consensus.cc:2229] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:26.953719 4630 raft_consensus.cc:2258] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:26.954443 4630 ts_tablet_manager.cc:1875] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:26.957301 4630 ts_tablet_manager.cc:1888] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.31
I20240503 14:01:26.957406 4630 log.cc:1196] T 8d6ad905b31541ffa6f2007d886b7643 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root/wals/8d6ad905b31541ffa6f2007d886b7643
I20240503 14:01:26.957926 3648 catalog_manager.cc:4830] TS f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053): tablet 8d6ad905b31541ffa6f2007d886b7643 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:26.973737 4160 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 1 LEARNER]: Advancing to term 2
I20240503 14:01:26.975152 4160 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 2 LEARNER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:26.975641 4571 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:26.977548 4571 raft_consensus.cc:1050] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: attempting to promote NON_VOTER 6d05d524f254418bb351065633a4606c to VOTER
W20240503 14:01:26.977703 4571 raft_consensus.cc:1053] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: Unable to promote non-voter 6d05d524f254418bb351065633a4606c: Illegal state: Leader has not yet committed an operation in its own term
I20240503 14:01:27.055936 4311 raft_consensus.cc:3037] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e [term 3 LEARNER]: Advancing to term 4
I20240503 14:01:27.138447 3762 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:27.138865 4571 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:27.139580 3914 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:27.140316 4626 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:27.143899 4627 raft_consensus.cc:1050] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: attempting to promote NON_VOTER 6d05d524f254418bb351065633a4606c to VOTER
I20240503 14:01:27.144443 4627 consensus_queue.cc:237] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:27.145932 3914 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:27.146296 4626 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:27.146358 4159 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 2 LEARNER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:27.147053 4626 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:27.147071 3762 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:27.147640 4626 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:27.148269 4624 raft_consensus.cc:1050] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: attempting to promote NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 to VOTER
I20240503 14:01:27.148682 4624 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 32, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:27.149611 4626 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:27.149909 4159 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:27.150136 3913 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240503 14:01:27.150669 4552 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 14:01:27.151731 3837 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240503 14:01:27.152216 3913 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:27.152900 4236 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 LEARNER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240503 14:01:27.153260 3660 catalog_manager.cc:5484] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 27 to 29, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:27.154048 3760 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, 6d05d524f254418bb351065633a4606c (127.0.99.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:27.156533 4640 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 14:01:27.156880 4552 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 14:01:27.158063 3840 consensus_queue.cc:237] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:27.158766 4640 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 32 to 34, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.159689 4159 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:27.159710 3913 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:27.160125 4600 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:27.160326 4627 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:27.160897 4236 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 32 to 34, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.161806 3659 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c reported cstate change: config changed from index 32 to 34, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "6d05d524f254418bb351065633a4606c" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:27.162068 3914 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 32 to 34, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.162186 3837 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 32 to 34, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.163249 4600 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:27.163745 3914 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:27.164892 3647 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:27.166026 3660 catalog_manager.cc:5484] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 29 to 30, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New cstate: current_term: 2 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:27.168566 4159 raft_consensus.cc:2932] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } }
I20240503 14:01:27.169217 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 with delete_type TABLET_DATA_TOMBSTONED (TS f71f8fdeb4004bb8b8012dc8b79ff8e1 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:52320
I20240503 14:01:27.169464 4630 tablet_replica.cc:330] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: stopping tablet replica
I20240503 14:01:27.169625 4630 raft_consensus.cc:2229] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:27.169816 4630 raft_consensus.cc:2258] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:27.171205 4159 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:27.172469 3837 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240503 14:01:27.172660 4630 ts_tablet_manager.cc:1875] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:27.173032 4639 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 14:01:27.174690 4624 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 4 LEADER]: Committing config change with OpId 4.35: config changed from index 34 to 35, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.175354 4630 ts_tablet_manager.cc:1888] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 14:01:27.175428 4630 log.cc:1196] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-0-root/wals/95a2d80baec84cefb6d8b5e0ca27e5d7
I20240503 14:01:27.176200 4236 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240503 14:01:27.177173 3648 catalog_manager.cc:4830] TS f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053): tablet 95a2d80baec84cefb6d8b5e0ca27e5d7 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:27.177675 3708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:27.177874 4624 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 14:01:27.178145 3837 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.180502 4236 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.181167 3659 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 34 to 35, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New cstate: current_term: 4 leader_uuid: "6d05d524f254418bb351065633a4606c" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.184432 3904 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with delete_type TABLET_DATA_TOMBSTONED (TS f6d3c1f17e6b4f42a813bef57d94fe87 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:47958
I20240503 14:01:27.184825 4643 tablet_replica.cc:330] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: stopping tablet replica
I20240503 14:01:27.184985 4643 raft_consensus.cc:2229] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:27.185214 4643 raft_consensus.cc:2258] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:27.185861 4643 ts_tablet_manager.cc:1875] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:27.188490 4643 ts_tablet_manager.cc:1888] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.34
I20240503 14:01:27.188602 4643 log.cc:1196] T b1bde81d4ddf4de3a2e16be46d97b1b5 P f6d3c1f17e6b4f42a813bef57d94fe87: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root/wals/b1bde81d4ddf4de3a2e16be46d97b1b5
I20240503 14:01:27.189114 3647 catalog_manager.cc:4830] TS f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425): tablet b1bde81d4ddf4de3a2e16be46d97b1b5 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:27.229892 3760 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:27.230382 4626 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:27.232820 4311 raft_consensus.cc:1261] T d121433c26cb4d89876ce434ba669006 P 06740b673b674aecadcbb0fa2bcac68e [term 2 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:27.233363 4626 consensus_queue.cc:1035] T d121433c26cb4d89876ce434ba669006 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:27.295431 3837 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:27.295882 4611 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:27.298731 3914 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:27.299198 4611 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:27.300184 4236 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 1 LEARNER]: Advancing to term 2
I20240503 14:01:27.301493 4236 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 LEARNER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:27.301919 4611 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:27.304205 4647 raft_consensus.cc:1050] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1: attempting to promote NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 to VOTER
I20240503 14:01:27.304659 4647 consensus_queue.cc:237] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:27.306159 3914 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:27.306159 3837 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:27.306591 4611 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:27.306747 4236 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 LEARNER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:27.306870 4611 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:27.307083 4648 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:27.308874 4652 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.309142 3837 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.310905 3914 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.311777 3657 catalog_manager.cc:5484] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 27 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.311784 4236 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.316306 3760 consensus_queue.cc:237] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:27.317682 3837 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:27.317699 4236 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:27.318269 4648 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:27.318506 4611 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:27.320313 4652 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.320804 4236 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.320813 3840 raft_consensus.cc:2932] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.322540 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:27.323619 3660 catalog_manager.cc:5484] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 29 to 30, VOTER f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) evicted. New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } }
I20240503 14:01:27.326220 3904 tablet_service.cc:1514] Processing DeleteTablet for tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 with delete_type TABLET_DATA_TOMBSTONED (TS f6d3c1f17e6b4f42a813bef57d94fe87 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:47958
I20240503 14:01:27.326496 4643 tablet_replica.cc:330] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: stopping tablet replica
I20240503 14:01:27.326651 4643 raft_consensus.cc:2229] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:27.326822 4643 raft_consensus.cc:2258] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:27.327443 4643 ts_tablet_manager.cc:1875] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:27.329998 4643 ts_tablet_manager.cc:1888] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240503 14:01:27.330112 4643 log.cc:1196] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f6d3c1f17e6b4f42a813bef57d94fe87: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-2-root/wals/95d4c1bc93314dbea0dc3a714c8c9ec6
I20240503 14:01:27.330614 3647 catalog_manager.cc:4830] TS f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425): tablet 95d4c1bc93314dbea0dc3a714c8c9ec6 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:27.333806 3760 consensus_queue.cc:237] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:27.334986 4311 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:27.335343 3914 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:27.335400 4648 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:27.335697 4652 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:27.337388 4611 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.337741 3914 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.337754 4311 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.339179 3648 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11893d5b97174d7389005a373091703b with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:27.340003 3657 catalog_manager.cc:5484] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: config changed from index 30 to 31, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New cstate: current_term: 3 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.342633 3827 tablet_service.cc:1514] Processing DeleteTablet for tablet 11893d5b97174d7389005a373091703b with delete_type TABLET_DATA_TOMBSTONED (TS 2d8cd9c1e8e2469888742770004dcb11 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:36456
I20240503 14:01:27.343050 4653 tablet_replica.cc:330] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: stopping tablet replica
I20240503 14:01:27.343233 4653 raft_consensus.cc:2229] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:27.343429 4653 raft_consensus.cc:2258] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:27.344133 4653 ts_tablet_manager.cc:1875] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:27.346824 4653 ts_tablet_manager.cc:1888] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20240503 14:01:27.346923 4653 log.cc:1196] T 11893d5b97174d7389005a373091703b P 2d8cd9c1e8e2469888742770004dcb11: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root/wals/11893d5b97174d7389005a373091703b
I20240503 14:01:27.347450 3647 catalog_manager.cc:4830] TS 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655): tablet 11893d5b97174d7389005a373091703b (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:27.557318 4655 raft_consensus.cc:1050] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67: attempting to promote NON_VOTER 06740b673b674aecadcbb0fa2bcac68e to VOTER
I20240503 14:01:27.557822 4655 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 32, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:27.559262 4311 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e [term 4 LEARNER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240503 14:01:27.559335 3914 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240503 14:01:27.559358 3840 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 4 index: 34. (index mismatch)
I20240503 14:01:27.559742 4654 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 14:01:27.563390 4654 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 14:01:27.563876 4654 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20240503 14:01:27.566051 4654 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 LEADER]: Committing config change with OpId 4.34: config changed from index 32 to 34, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.566543 3840 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 32 to 34, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.569401 3660 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: config changed from index 32 to 34, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.570955 3914 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 32 to 34, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.572211 4311 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e [term 4 FOLLOWER]: Committing config change with OpId 4.34: config changed from index 32 to 34, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.575942 4235 consensus_queue.cc:237] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 4.34, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:27.577119 3914 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240503 14:01:27.577138 4311 raft_consensus.cc:1261] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e [term 4 FOLLOWER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 4 index: 34. Preceding OpId from leader: term: 4 index: 35. (index mismatch)
I20240503 14:01:27.577549 4592 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 14:01:27.577737 4655 consensus_queue.cc:1035] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20240503 14:01:27.579434 4657 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 LEADER]: Committing config change with OpId 4.35: config changed from index 34 to 35, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.579763 4311 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P 06740b673b674aecadcbb0fa2bcac68e [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.580008 3914 raft_consensus.cc:2932] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Committing config change with OpId 4.35: config changed from index 34 to 35, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.581161 3706 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0171f05f97a24f97ab8732925b360c82 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:27.582232 3660 catalog_manager.cc:5484] T 0171f05f97a24f97ab8732925b360c82 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: config changed from index 34 to 35, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New cstate: current_term: 4 leader_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:27.584455 3827 tablet_service.cc:1514] Processing DeleteTablet for tablet 0171f05f97a24f97ab8732925b360c82 with delete_type TABLET_DATA_TOMBSTONED (TS 2d8cd9c1e8e2469888742770004dcb11 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:36456
I20240503 14:01:27.584714 4653 tablet_replica.cc:330] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: stopping tablet replica
I20240503 14:01:27.584853 4653 raft_consensus.cc:2229] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:27.585047 4653 raft_consensus.cc:2258] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:27.585641 4653 ts_tablet_manager.cc:1875] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:27.587415 4653 ts_tablet_manager.cc:1888] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.34
I20240503 14:01:27.587487 4653 log.cc:1196] T 0171f05f97a24f97ab8732925b360c82 P 2d8cd9c1e8e2469888742770004dcb11: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root/wals/0171f05f97a24f97ab8732925b360c82
I20240503 14:01:27.587982 3647 catalog_manager.cc:4830] TS 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655): tablet 0171f05f97a24f97ab8732925b360c82 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:28.458927 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:28.460898 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
new_leader_uuid: "e4f6fd3611bf4f0c812a03bde535fb67"
from {username='slave'} at 127.0.0.1:52372
I20240503 14:01:28.461084 3760 raft_consensus.cc:595] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership to TS e4f6fd3611bf4f0c812a03bde535fb67
I20240503 14:01:28.461992 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8af6f6a1be8f43a693011cfa1c022f08"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:28.462146 3840 raft_consensus.cc:595] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Received request to transfer leadership to TS f71f8fdeb4004bb8b8012dc8b79ff8e1
I20240503 14:01:28.462868 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:28.463003 3840 raft_consensus.cc:595] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Received request to transfer leadership to TS f6d3c1f17e6b4f42a813bef57d94fe87
I20240503 14:01:28.463281 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240503 14:01:28.463452 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:28.575642 4627 raft_consensus.cc:979] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:28.575786 4627 raft_consensus.cc:1067] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Signalling peer f71f8fdeb4004bb8b8012dc8b79ff8e1 to start an election
I20240503 14:01:28.576290 3760 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8af6f6a1be8f43a693011cfa1c022f08"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
from {username='slave'} at 127.0.0.1:52346
I20240503 14:01:28.576442 3760 raft_consensus.cc:482] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:28.576507 3760 raft_consensus.cc:3037] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:28.577606 3760 raft_consensus.cc:504] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:28.577982 3760 leader_election.cc:288] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:28.578409 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af6f6a1be8f43a693011cfa1c022f08" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:28.578480 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af6f6a1be8f43a693011cfa1c022f08" candidate_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:28.578529 3840 raft_consensus.cc:3032] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:28.578570 3840 raft_consensus.cc:729] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:28.578593 3914 raft_consensus.cc:3037] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:28.578743 3840 consensus_queue.cc:260] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:28.578907 3840 raft_consensus.cc:3037] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:28.579548 3914 raft_consensus.cc:2454] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:28.579818 3840 raft_consensus.cc:2454] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f71f8fdeb4004bb8b8012dc8b79ff8e1 in term 2.
I20240503 14:01:28.579979 3729 leader_election.cc:302] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6d3c1f17e6b4f42a813bef57d94fe87, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:28.580193 4648 raft_consensus.cc:2781] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:28.580384 4648 raft_consensus.cc:686] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Leader. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:28.580533 4648 consensus_queue.cc:237] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
I20240503 14:01:28.582254 3659 catalog_manager.cc:5484] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: term changed from 1 to 2, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1). New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:28.585935 4159 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 4.35, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:28.587514 3840 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240503 14:01:28.587533 4236 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 35. Preceding OpId from leader: term: 4 index: 36. (index mismatch)
I20240503 14:01:28.588083 4670 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 14:01:28.588325 4669 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20240503 14:01:28.590157 4670 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 4 LEADER]: Committing config change with OpId 4.36: config changed from index 35 to 36, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:28.590603 3840 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
W20240503 14:01:28.591187 4124 consensus_peers.cc:485] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1bde81d4ddf4de3a2e16be46d97b1b5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:28.592625 4236 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Committing config change with OpId 4.36: config changed from index 35 to 36, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:28.593137 3659 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c reported cstate change: config changed from index 35 to 36, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New cstate: current_term: 4 leader_uuid: "6d05d524f254418bb351065633a4606c" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:01:28.593286 3709 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20240503 14:01:28.596130 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 71dd0142015e4fb791413d533b9ccb55, TS f6d3c1f17e6b4f42a813bef57d94fe87 -> TS 6d05d524f254418bb351065633a4606c move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
W20240503 14:01:28.596243 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 71dd0142015e4fb791413d533b9ccb55, TS f6d3c1f17e6b4f42a813bef57d94fe87 -> TS 6d05d524f254418bb351065633a4606c move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
W20240503 14:01:28.597995 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet cc6eba6dbb2c4efda6d24f9ee9d6e8f7, TS f71f8fdeb4004bb8b8012dc8b79ff8e1 -> TS e4f6fd3611bf4f0c812a03bde535fb67 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
W20240503 14:01:28.598071 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet cc6eba6dbb2c4efda6d24f9ee9d6e8f7, TS f71f8fdeb4004bb8b8012dc8b79ff8e1 -> TS e4f6fd3611bf4f0c812a03bde535fb67 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
W20240503 14:01:28.599752 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8af6f6a1be8f43a693011cfa1c022f08, TS f6d3c1f17e6b4f42a813bef57d94fe87 -> TS 6d05d524f254418bb351065633a4606c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
W20240503 14:01:28.599820 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 8af6f6a1be8f43a693011cfa1c022f08, TS f6d3c1f17e6b4f42a813bef57d94fe87 -> TS 6d05d524f254418bb351065633a4606c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
W20240503 14:01:28.611423 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6b7539be4b0c41d1accce62f9abbc9e9, TS 2d8cd9c1e8e2469888742770004dcb11 -> TS e4f6fd3611bf4f0c812a03bde535fb67 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
W20240503 14:01:28.611527 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 6b7539be4b0c41d1accce62f9abbc9e9, TS 2d8cd9c1e8e2469888742770004dcb11 -> TS e4f6fd3611bf4f0c812a03bde535fb67 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
W20240503 14:01:28.613772 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 11c78a8dc746409aa487dcd7ab58cc82, TS f71f8fdeb4004bb8b8012dc8b79ff8e1 -> TS 06740b673b674aecadcbb0fa2bcac68e move failed, destination replica disappeared from tablet's Raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
W20240503 14:01:28.613862 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 11c78a8dc746409aa487dcd7ab58cc82, TS f71f8fdeb4004bb8b8012dc8b79ff8e1 -> TS 06740b673b674aecadcbb0fa2bcac68e move failed, destination replica disappeared from tablet's Raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:28.859023 4571 raft_consensus.cc:979] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:28.859148 4571 raft_consensus.cc:1067] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Signalling peer f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:28.859511 3914 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:47972
I20240503 14:01:28.859699 3914 raft_consensus.cc:482] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:28.859767 3914 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:01:28.860911 3914 raft_consensus.cc:504] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:28.861285 3914 leader_election.cc:288] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 5 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), 6d05d524f254418bb351065633a4606c (127.0.99.4:42471)
I20240503 14:01:28.861526 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:28.861644 3840 raft_consensus.cc:3032] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Stepping down as leader of term 4
I20240503 14:01:28.861701 3840 raft_consensus.cc:729] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:28.861784 4160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "6d05d524f254418bb351065633a4606c"
I20240503 14:01:28.861869 3840 consensus_queue.cc:260] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:28.861876 4160 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:01:28.862078 3840 raft_consensus.cc:3037] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Advancing to term 5
I20240503 14:01:28.863108 4160 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 5.
I20240503 14:01:28.863108 3840 raft_consensus.cc:2454] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 5.
I20240503 14:01:28.863399 3881 leader_election.cc:302] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d05d524f254418bb351065633a4606c, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:28.863617 4564 raft_consensus.cc:2781] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 5 FOLLOWER]: Leader election won for term 5
I20240503 14:01:28.863821 4564 raft_consensus.cc:686] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 5 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:28.863976 4564 consensus_queue.cc:237] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:28.865592 3657 catalog_manager.cc:5484] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 4 to 5, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 5 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:28.909729 4663 raft_consensus.cc:979] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1: : Instructing follower e4f6fd3611bf4f0c812a03bde535fb67 to start an election
I20240503 14:01:28.909883 4663 raft_consensus.cc:1067] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Signalling peer e4f6fd3611bf4f0c812a03bde535fb67 to start an election
I20240503 14:01:28.910256 4236 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55"
dest_uuid: "e4f6fd3611bf4f0c812a03bde535fb67"
from {username='slave'} at 127.0.0.1:49596
I20240503 14:01:28.910392 4236 raft_consensus.cc:482] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:28.910451 4236 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:28.911360 4236 raft_consensus.cc:504] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:28.911854 4236 leader_election.cc:288] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [CANDIDATE]: Term 3 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:28.912182 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55" candidate_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:28.912211 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71dd0142015e4fb791413d533b9ccb55" candidate_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:28.912319 3914 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:28.913511 3914 raft_consensus.cc:2454] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f6fd3611bf4f0c812a03bde535fb67 in term 3.
I20240503 14:01:28.912336 3760 raft_consensus.cc:3032] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:01:28.913681 3760 raft_consensus.cc:729] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:28.913817 4201 leader_election.cc:302] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4f6fd3611bf4f0c812a03bde535fb67, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:28.913844 3760 consensus_queue.cc:260] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:28.914014 4662 raft_consensus.cc:2781] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:01:28.914028 3760 raft_consensus.cc:3037] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:28.914294 4662 raft_consensus.cc:686] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 LEADER]: Becoming Leader. State: Replica: e4f6fd3611bf4f0c812a03bde535fb67, State: Running, Role: LEADER
I20240503 14:01:28.914446 4662 consensus_queue.cc:237] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:28.915020 3760 raft_consensus.cc:2454] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4f6fd3611bf4f0c812a03bde535fb67 in term 3.
I20240503 14:01:28.916024 3657 catalog_manager.cc:5484] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 reported cstate change: term changed from 2 to 3, leader changed from f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) to e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5). New cstate: current_term: 3 leader_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:28.989923 3914 raft_consensus.cc:1261] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 14:01:28.990309 4663 consensus_queue.cc:1035] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:01:28.993099 3840 raft_consensus.cc:1261] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20240503 14:01:28.993499 4648 consensus_queue.cc:1035] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240503 14:01:29.197949 4685 ts_tablet_manager.cc:902] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: Initiating tablet copy from peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471)
I20240503 14:01:29.198586 4685 tablet_copy_client.cc:308] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.4:42471
I20240503 14:01:29.202657 4170 tablet_copy_service.cc:140] P 6d05d524f254418bb351065633a4606c: Received BeginTabletCopySession request for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 from peer 06740b673b674aecadcbb0fa2bcac68e ({username='slave'} at 127.0.0.1:35080)
I20240503 14:01:29.202805 4170 tablet_copy_service.cc:161] P 6d05d524f254418bb351065633a4606c: Beginning new tablet copy session on tablet b1bde81d4ddf4de3a2e16be46d97b1b5 from peer 06740b673b674aecadcbb0fa2bcac68e at {username='slave'} at 127.0.0.1:35080: session id = 06740b673b674aecadcbb0fa2bcac68e-b1bde81d4ddf4de3a2e16be46d97b1b5
I20240503 14:01:29.203930 4170 tablet_copy_source_session.cc:205] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:29.204762 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1bde81d4ddf4de3a2e16be46d97b1b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:29.207144 4685 tablet_copy_client.cc:783] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:29.207324 4685 tablet_copy_client.cc:647] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:29.208374 4689 tablet_copy_client.cc:1002] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:29.245637 3840 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P 2d8cd9c1e8e2469888742770004dcb11 [term 5 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 14:01:29.246054 4564 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 14:01:29.248754 4160 raft_consensus.cc:1261] T 8d6ad905b31541ffa6f2007d886b7643 P 6d05d524f254418bb351065633a4606c [term 5 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20240503 14:01:29.249236 4693 consensus_queue.cc:1035] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240503 14:01:29.307446 3760 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:29.307878 4662 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:29.310429 3914 raft_consensus.cc:1261] T 71dd0142015e4fb791413d533b9ccb55 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer e4f6fd3611bf4f0c812a03bde535fb67: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:29.311002 4678 consensus_queue.cc:1035] T 71dd0142015e4fb791413d533b9ccb55 P e4f6fd3611bf4f0c812a03bde535fb67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:30.463732 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:30.465533 3914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "11c78a8dc746409aa487dcd7ab58cc82"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
mode: GRACEFUL
new_leader_uuid: "6d05d524f254418bb351065633a4606c"
from {username='slave'} at 127.0.0.1:47982
I20240503 14:01:30.465703 3914 raft_consensus.cc:595] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Received request to transfer leadership to TS 6d05d524f254418bb351065633a4606c
I20240503 14:01:30.466394 3760 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6"
dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
mode: GRACEFUL
new_leader_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:52372
I20240503 14:01:30.466519 3760 raft_consensus.cc:595] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Received request to transfer leadership to TS 2d8cd9c1e8e2469888742770004dcb11
I20240503 14:01:30.466866 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 2
I20240503 14:01:30.467003 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:30.841229 4648 raft_consensus.cc:979] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1: : Instructing follower 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:30.841360 4648 raft_consensus.cc:1067] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Signalling peer 2d8cd9c1e8e2469888742770004dcb11 to start an election
I20240503 14:01:30.841780 3840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:36472
I20240503 14:01:30.841939 3840 raft_consensus.cc:482] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:30.842012 3840 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:30.843129 3840 raft_consensus.cc:504] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:30.843554 3840 leader_election.cc:288] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 3 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127)
I20240503 14:01:30.843917 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:30.844058 3760 raft_consensus.cc:3032] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:01:30.844103 3760 raft_consensus.cc:729] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f71f8fdeb4004bb8b8012dc8b79ff8e1, State: Running, Role: LEADER
I20240503 14:01:30.844254 3760 consensus_queue.cc:260] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:30.844414 3760 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:30.845297 3760 raft_consensus.cc:2454] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 3.
I20240503 14:01:30.843876 4236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95d4c1bc93314dbea0dc3a714c8c9ec6" candidate_uuid: "2d8cd9c1e8e2469888742770004dcb11" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "e4f6fd3611bf4f0c812a03bde535fb67"
I20240503 14:01:30.845566 4236 raft_consensus.cc:3037] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:30.845649 3806 leader_election.cc:302] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f71f8fdeb4004bb8b8012dc8b79ff8e1; no voters:
I20240503 14:01:30.845834 4627 raft_consensus.cc:2781] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:01:30.845999 4627 raft_consensus.cc:686] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 LEADER]: Becoming Leader. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:30.846117 4627 consensus_queue.cc:237] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } }
I20240503 14:01:30.846427 4236 raft_consensus.cc:2454] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d8cd9c1e8e2469888742770004dcb11 in term 3.
I20240503 14:01:30.847805 3657 catalog_manager.cc:5484] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 reported cstate change: term changed from 2 to 3, leader changed from f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1) to 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2). New cstate: current_term: 3 leader_uuid: "2d8cd9c1e8e2469888742770004dcb11" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:30.991179 4705 raft_consensus.cc:979] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87: : Instructing follower 6d05d524f254418bb351065633a4606c to start an election
I20240503 14:01:30.991298 4705 raft_consensus.cc:1067] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Signalling peer 6d05d524f254418bb351065633a4606c to start an election
I20240503 14:01:30.991704 4159 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "11c78a8dc746409aa487dcd7ab58cc82"
dest_uuid: "6d05d524f254418bb351065633a4606c"
from {username='slave'} at 127.0.0.1:48032
I20240503 14:01:30.991864 4159 raft_consensus.cc:482] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:30.991933 4159 raft_consensus.cc:3037] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:30.993162 4159 raft_consensus.cc:504] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:30.993677 4159 leader_election.cc:288] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [CANDIDATE]: Term 2 election: Requested vote from peers f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053), f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
I20240503 14:01:30.993939 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c78a8dc746409aa487dcd7ab58cc82" candidate_uuid: "6d05d524f254418bb351065633a4606c" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
I20240503 14:01:30.994058 3914 raft_consensus.cc:3032] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:01:30.994100 3914 raft_consensus.cc:729] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:30.994259 3914 consensus_queue.cc:260] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:30.994443 3914 raft_consensus.cc:3037] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:30.995599 3914 raft_consensus.cc:2454] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d05d524f254418bb351065633a4606c in term 2.
I20240503 14:01:30.995935 4125 leader_election.cc:302] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d05d524f254418bb351065633a4606c, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:30.996116 4704 raft_consensus.cc:2781] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:01:30.996351 4704 raft_consensus.cc:686] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 2 LEADER]: Becoming Leader. State: Replica: 6d05d524f254418bb351065633a4606c, State: Running, Role: LEADER
I20240503 14:01:30.996519 4704 consensus_queue.cc:237] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:30.998118 3760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c78a8dc746409aa487dcd7ab58cc82" candidate_uuid: "6d05d524f254418bb351065633a4606c" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1"
I20240503 14:01:30.998256 3760 raft_consensus.cc:3037] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:01:30.998399 3657 catalog_manager.cc:5484] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c reported cstate change: term changed from 1 to 2, leader changed from f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3) to 6d05d524f254418bb351065633a4606c (127.0.99.4). New cstate: current_term: 2 leader_uuid: "6d05d524f254418bb351065633a4606c" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:30.999387 3760 raft_consensus.cc:2454] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d05d524f254418bb351065633a4606c in term 2.
I20240503 14:01:31.208716 4685 tablet_copy_client.cc:523] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:01:31.210692 4685 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap starting.
I20240503 14:01:31.238746 4685 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap replayed 1/1 log segments. Stats: ops{read=36 overwritten=0 applied=36 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:01:31.239096 4685 tablet_bootstrap.cc:492] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: Bootstrap complete.
I20240503 14:01:31.239321 4685 ts_tablet_manager.cc:1367] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: Time spent bootstrapping tablet: real 0.029s user 0.027s sys 0.000s
I20240503 14:01:31.239881 4685 raft_consensus.cc:348] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:31.240020 4685 raft_consensus.cc:729] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 06740b673b674aecadcbb0fa2bcac68e, State: Initialized, Role: LEARNER
I20240503 14:01:31.240128 4685 consensus_queue.cc:260] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 36, Last appended: 4.36, Last appended by leader: 36, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:31.240487 4685 ts_tablet_manager.cc:1398] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240503 14:01:31.240911 4170 tablet_copy_service.cc:342] P 6d05d524f254418bb351065633a4606c: Request end of tablet copy session 06740b673b674aecadcbb0fa2bcac68e-b1bde81d4ddf4de3a2e16be46d97b1b5 received from {username='slave'} at 127.0.0.1:35080
I20240503 14:01:31.240983 4170 tablet_copy_service.cc:433] P 6d05d524f254418bb351065633a4606c: ending tablet copy session 06740b673b674aecadcbb0fa2bcac68e-b1bde81d4ddf4de3a2e16be46d97b1b5 on tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with peer 06740b673b674aecadcbb0fa2bcac68e
I20240503 14:01:31.262831 3760 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:31.263255 4627 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:31.265728 4236 raft_consensus.cc:1261] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P e4f6fd3611bf4f0c812a03bde535fb67 [term 3 FOLLOWER]: Refusing update from remote peer 2d8cd9c1e8e2469888742770004dcb11: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:31.266537 4627 consensus_queue.cc:1035] T 95d4c1bc93314dbea0dc3a714c8c9ec6 P 2d8cd9c1e8e2469888742770004dcb11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:31.382624 3914 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:31.383116 4704 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:31.385824 3760 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240503 14:01:31.386312 4715 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240503 14:01:31.444216 4311 raft_consensus.cc:1203] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e [term 4 LEARNER]: Deduplicated request from leader. Original: 4.35->[4.36-4.36] Dedup: 4.36->[]
I20240503 14:01:31.921964 4729 raft_consensus.cc:1050] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c: attempting to promote NON_VOTER 06740b673b674aecadcbb0fa2bcac68e to VOTER
I20240503 14:01:31.922626 4729 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 4.36, Last appended by leader: 32, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:31.924124 4311 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e [term 4 LEARNER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240503 14:01:31.924124 4236 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240503 14:01:31.924124 3840 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 36. Preceding OpId from leader: term: 4 index: 37. (index mismatch)
I20240503 14:01:31.924615 4730 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 14:01:31.924927 4732 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 14:01:31.925209 4729 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240503 14:01:31.927606 4729 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 4 LEADER]: Committing config change with OpId 4.37: config changed from index 36 to 37, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:31.927973 4236 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:31.928016 4311 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:31.930586 3657 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c reported cstate change: config changed from index 36 to 37, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "6d05d524f254418bb351065633a4606c" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:31.931944 3840 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Committing config change with OpId 4.37: config changed from index 36 to 37, 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:31.934577 4160 consensus_queue.cc:237] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 4.37, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:31.936336 4236 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 37. Preceding OpId from leader: term: 4 index: 38. (index mismatch)
I20240503 14:01:31.936336 4311 raft_consensus.cc:1261] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e [term 4 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 4 index: 37. Preceding OpId from leader: term: 4 index: 38. (index mismatch)
I20240503 14:01:31.936738 4730 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 14:01:31.936970 4704 consensus_queue.cc:1035] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20240503 14:01:31.938463 4729 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c [term 4 LEADER]: Committing config change with OpId 4.38: config changed from index 37 to 38, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:31.938958 4311 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 06740b673b674aecadcbb0fa2bcac68e [term 4 FOLLOWER]: Committing config change with OpId 4.38: config changed from index 37 to 38, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:31.939968 4236 raft_consensus.cc:2932] T b1bde81d4ddf4de3a2e16be46d97b1b5 P e4f6fd3611bf4f0c812a03bde535fb67 [term 4 FOLLOWER]: Committing config change with OpId 4.38: config changed from index 37 to 38, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } }
I20240503 14:01:31.941332 3660 catalog_manager.cc:5484] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 6d05d524f254418bb351065633a4606c reported cstate change: config changed from index 37 to 38, VOTER 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) evicted. New cstate: current_term: 4 leader_uuid: "6d05d524f254418bb351065633a4606c" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:01:31.942098 3708 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with cas_config_opid_index 37: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:01:31.944268 3827 tablet_service.cc:1514] Processing DeleteTablet for tablet b1bde81d4ddf4de3a2e16be46d97b1b5 with delete_type TABLET_DATA_TOMBSTONED (TS 2d8cd9c1e8e2469888742770004dcb11 not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:36456
I20240503 14:01:31.944722 4739 tablet_replica.cc:330] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: stopping tablet replica
I20240503 14:01:31.944875 4739 raft_consensus.cc:2229] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Raft consensus shutting down.
I20240503 14:01:31.945060 4739 raft_consensus.cc:2258] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11 [term 4 FOLLOWER]: Raft consensus is shut down!
I20240503 14:01:31.945708 4739 ts_tablet_manager.cc:1875] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:01:31.948000 4739 ts_tablet_manager.cc:1888] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.37
I20240503 14:01:31.948094 4739 log.cc:1196] T b1bde81d4ddf4de3a2e16be46d97b1b5 P 2d8cd9c1e8e2469888742770004dcb11: Deleting WAL directory at /tmp/dist-test-task26heU2/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714744785267298-396-0/minicluster-data/ts-1-root/wals/b1bde81d4ddf4de3a2e16be46d97b1b5
I20240503 14:01:31.948689 3647 catalog_manager.cc:4830] TS 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655): tablet b1bde81d4ddf4de3a2e16be46d97b1b5 (table test-workload [id=841e0752055246b7a9dbb6fb916cdab6]) successfully deleted
I20240503 14:01:32.467403 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:32.469642 3840 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7"
dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
mode: GRACEFUL
new_leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:36484
I20240503 14:01:32.469836 3840 raft_consensus.cc:595] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Received request to transfer leadership to TS f6d3c1f17e6b4f42a813bef57d94fe87
I20240503 14:01:32.470242 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 14:01:32.470397 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
W20240503 14:01:32.946707 3709 auto_rebalancer.cc:253] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
W20240503 14:01:32.947826 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 95a2d80baec84cefb6d8b5e0ca27e5d7, TS f6d3c1f17e6b4f42a813bef57d94fe87 -> TS e4f6fd3611bf4f0c812a03bde535fb67 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
W20240503 14:01:32.947906 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 95a2d80baec84cefb6d8b5e0ca27e5d7, TS f6d3c1f17e6b4f42a813bef57d94fe87 -> TS e4f6fd3611bf4f0c812a03bde535fb67 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
W20240503 14:01:32.948870 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6b7539be4b0c41d1accce62f9abbc9e9, TS f71f8fdeb4004bb8b8012dc8b79ff8e1 -> TS 6d05d524f254418bb351065633a4606c move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
W20240503 14:01:32.948942 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 6b7539be4b0c41d1accce62f9abbc9e9, TS f71f8fdeb4004bb8b8012dc8b79ff8e1 -> TS 6d05d524f254418bb351065633a4606c move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
W20240503 14:01:32.949887 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 11c78a8dc746409aa487dcd7ab58cc82, TS f6d3c1f17e6b4f42a813bef57d94fe87 -> TS 06740b673b674aecadcbb0fa2bcac68e move failed, destination replica disappeared from tablet's Raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
W20240503 14:01:32.949949 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 11c78a8dc746409aa487dcd7ab58cc82, TS f6d3c1f17e6b4f42a813bef57d94fe87 -> TS 06740b673b674aecadcbb0fa2bcac68e move failed, destination replica disappeared from tablet's Raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
W20240503 14:01:32.950820 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8af6f6a1be8f43a693011cfa1c022f08, TS 2d8cd9c1e8e2469888742770004dcb11 -> TS 6d05d524f254418bb351065633a4606c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
W20240503 14:01:32.950875 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet 8af6f6a1be8f43a693011cfa1c022f08, TS 2d8cd9c1e8e2469888742770004dcb11 -> TS 6d05d524f254418bb351065633a4606c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } }
W20240503 14:01:32.951761 3709 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d121433c26cb4d89876ce434ba669006, TS f71f8fdeb4004bb8b8012dc8b79ff8e1 -> TS e4f6fd3611bf4f0c812a03bde535fb67 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
W20240503 14:01:32.951817 3709 auto_rebalancer.cc:263] scheduled replica move failed to complete: Incomplete: tablet d121433c26cb4d89876ce434ba669006, TS f71f8fdeb4004bb8b8012dc8b79ff8e1 -> TS e4f6fd3611bf4f0c812a03bde535fb67 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } }
I20240503 14:01:33.022979 4728 raft_consensus.cc:979] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11: : Instructing follower f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:33.023092 4728 raft_consensus.cc:1067] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Signalling peer f6d3c1f17e6b4f42a813bef57d94fe87 to start an election
I20240503 14:01:33.023592 3914 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
from {username='slave'} at 127.0.0.1:47972
I20240503 14:01:33.023741 3914 raft_consensus.cc:482] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:01:33.023808 3914 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:33.024703 3914 raft_consensus.cc:504] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:33.025169 3914 leader_election.cc:288] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 3 election: Requested vote from peers 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2:40655), 6d05d524f254418bb351065633a4606c (127.0.99.4:42471)
I20240503 14:01:33.025563 3840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "2d8cd9c1e8e2469888742770004dcb11"
I20240503 14:01:33.025534 4159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95a2d80baec84cefb6d8b5e0ca27e5d7" candidate_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "6d05d524f254418bb351065633a4606c"
I20240503 14:01:33.025669 3840 raft_consensus.cc:3032] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Stepping down as leader of term 2
I20240503 14:01:33.025671 4159 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:33.025740 3840 raft_consensus.cc:729] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2d8cd9c1e8e2469888742770004dcb11, State: Running, Role: LEADER
I20240503 14:01:33.025910 3840 consensus_queue.cc:260] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:33.026088 3840 raft_consensus.cc:3037] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:01:33.026866 4159 raft_consensus.cc:2454] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 3.
I20240503 14:01:33.026957 3840 raft_consensus.cc:2454] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d3c1f17e6b4f42a813bef57d94fe87 in term 3.
I20240503 14:01:33.027241 3880 leader_election.cc:302] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d8cd9c1e8e2469888742770004dcb11, f6d3c1f17e6b4f42a813bef57d94fe87; no voters:
I20240503 14:01:33.027442 4705 raft_consensus.cc:2781] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:01:33.027650 4705 raft_consensus.cc:686] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 LEADER]: Becoming Leader. State: Replica: f6d3c1f17e6b4f42a813bef57d94fe87, State: Running, Role: LEADER
I20240503 14:01:33.027806 4705 consensus_queue.cc:237] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } }
I20240503 14:01:33.029431 3657 catalog_manager.cc:5484] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: term changed from 2 to 3, leader changed from 2d8cd9c1e8e2469888742770004dcb11 (127.0.99.2) to f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3). New cstate: current_term: 3 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:33.615324 4159 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 6d05d524f254418bb351065633a4606c [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:33.615759 4705 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:33.618522 3840 raft_consensus.cc:1261] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P 2d8cd9c1e8e2469888742770004dcb11 [term 3 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240503 14:01:33.619199 4705 consensus_queue.cc:1035] T 95a2d80baec84cefb6d8b5e0ca27e5d7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:33.955669 4159 consensus_queue.cc:237] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
W20240503 14:01:33.957387 4125 consensus_peers.cc:485] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11c78a8dc746409aa487dcd7ab58cc82. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:33.957408 3760 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:33.957388 3914 raft_consensus.cc:1261] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer 6d05d524f254418bb351065633a4606c: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240503 14:01:33.957926 4755 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:33.958160 4754 consensus_queue.cc:1035] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240503 14:01:33.960116 4754 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:33.960618 3914 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:33.960813 3760 raft_consensus.cc:2932] T 11c78a8dc746409aa487dcd7ab58cc82 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:33.963192 3657 catalog_manager.cc:5484] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c reported cstate change: config changed from index 27 to 29, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New cstate: current_term: 2 leader_uuid: "6d05d524f254418bb351065633a4606c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:33.964100 3760 consensus_queue.cc:237] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:33.965747 3840 raft_consensus.cc:1261] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:01:33.965785 3914 raft_consensus.cc:1261] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240503 14:01:33.966225 4727 consensus_queue.cc:1035] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240503 14:01:33.966620 4727 consensus_queue.cc:1035] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20240503 14:01:33.967474 3730 consensus_peers.cc:485] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 -> Peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471): Couldn't send request to peer 6d05d524f254418bb351065633a4606c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8af6f6a1be8f43a693011cfa1c022f08. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:33.968621 4727 raft_consensus.cc:2932] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:33.968967 3840 raft_consensus.cc:2932] T 8af6f6a1be8f43a693011cfa1c022f08 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:33.969049 3914 raft_consensus.cc:2932] T 8af6f6a1be8f43a693011cfa1c022f08 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:33.971304 3660 catalog_manager.cc:5484] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index -1 to 27, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New cstate: current_term: 2 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:33.971606 3760 consensus_queue.cc:237] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } }
I20240503 14:01:33.973151 3914 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 14:01:33.973179 4311 raft_consensus.cc:1261] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 3 FOLLOWER]: Refusing update from remote peer f71f8fdeb4004bb8b8012dc8b79ff8e1: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240503 14:01:33.973506 4763 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240503 14:01:33.973752 4727 consensus_queue.cc:1035] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240503 14:01:33.974510 3730 consensus_peers.cc:485] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 -> Peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471): Couldn't send request to peer 6d05d524f254418bb351065633a4606c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11893d5b97174d7389005a373091703b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:33.975482 4763 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:33.975951 4311 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P 06740b673b674aecadcbb0fa2bcac68e [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:33.977983 3914 raft_consensus.cc:2932] T 11893d5b97174d7389005a373091703b P f6d3c1f17e6b4f42a813bef57d94fe87 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } } }
I20240503 14:01:33.978149 3659 catalog_manager.cc:5484] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1 reported cstate change: config changed from index 31 to 32, NON_VOTER 6d05d524f254418bb351065633a4606c (127.0.99.4) added. New cstate: current_term: 3 leader_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d05d524f254418bb351065633a4606c" member_type: NON_VOTER last_known_addr { host: "127.0.99.4" port: 42471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:33.978472 4311 consensus_queue.cc:237] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } }
I20240503 14:01:33.980013 3760 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Refusing update from remote peer 06740b673b674aecadcbb0fa2bcac68e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:33.980343 4615 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:33.980876 3840 raft_consensus.cc:1261] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Refusing update from remote peer 06740b673b674aecadcbb0fa2bcac68e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240503 14:01:33.981547 4615 consensus_queue.cc:1035] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240503 14:01:33.984122 4775 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:33.984567 3760 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
I20240503 14:01:33.984616 3840 raft_consensus.cc:2932] T 6b7539be4b0c41d1accce62f9abbc9e9 P 2d8cd9c1e8e2469888742770004dcb11 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } } }
W20240503 14:01:33.987159 4276 consensus_peers.cc:485] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e -> Peer e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5:37127): Couldn't send request to peer e4f6fd3611bf4f0c812a03bde535fb67. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b7539be4b0c41d1accce62f9abbc9e9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:33.987162 3660 catalog_manager.cc:5484] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e reported cstate change: config changed from index 28 to 30, NON_VOTER e4f6fd3611bf4f0c812a03bde535fb67 (127.0.99.5) added. New cstate: current_term: 2 leader_uuid: "06740b673b674aecadcbb0fa2bcac68e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4f6fd3611bf4f0c812a03bde535fb67" member_type: NON_VOTER last_known_addr { host: "127.0.99.5" port: 37127 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:33.988924 3914 consensus_queue.cc:237] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } }
I20240503 14:01:33.990836 3840 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 5 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
W20240503 14:01:33.990938 3879 consensus_peers.cc:485] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 -> Peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057): Couldn't send request to peer 06740b673b674aecadcbb0fa2bcac68e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc6eba6dbb2c4efda6d24f9ee9d6e8f7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:01:33.991248 4748 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:33.991722 3760 raft_consensus.cc:1261] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 5 FOLLOWER]: Refusing update from remote peer f6d3c1f17e6b4f42a813bef57d94fe87: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20240503 14:01:33.992116 4777 consensus_queue.cc:1035] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240503 14:01:33.993196 4748 raft_consensus.cc:2932] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 5 LEADER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:33.993575 3840 raft_consensus.cc:2932] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 2d8cd9c1e8e2469888742770004dcb11 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:33.994167 3760 raft_consensus.cc:2932] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f71f8fdeb4004bb8b8012dc8b79ff8e1 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } } }
I20240503 14:01:33.995736 3659 catalog_manager.cc:5484] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P f6d3c1f17e6b4f42a813bef57d94fe87 reported cstate change: config changed from index -1 to 31, NON_VOTER 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6) added. New cstate: current_term: 5 leader_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "f71f8fdeb4004bb8b8012dc8b79ff8e1" member_type: VOTER last_known_addr { host: "127.0.99.1" port: 38053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d8cd9c1e8e2469888742770004dcb11" member_type: VOTER last_known_addr { host: "127.0.99.2" port: 40655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87" member_type: VOTER last_known_addr { host: "127.0.99.3" port: 43425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06740b673b674aecadcbb0fa2bcac68e" member_type: NON_VOTER last_known_addr { host: "127.0.99.6" port: 33057 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:01:34.400717 4780 ts_tablet_manager.cc:902] T 11c78a8dc746409aa487dcd7ab58cc82 P e4f6fd3611bf4f0c812a03bde535fb67: Initiating tablet copy from peer 6d05d524f254418bb351065633a4606c (127.0.99.4:42471)
I20240503 14:01:34.401371 4780 tablet_copy_client.cc:308] T 11c78a8dc746409aa487dcd7ab58cc82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.4:42471
I20240503 14:01:34.405320 4170 tablet_copy_service.cc:140] P 6d05d524f254418bb351065633a4606c: Received BeginTabletCopySession request for tablet 11c78a8dc746409aa487dcd7ab58cc82 from peer e4f6fd3611bf4f0c812a03bde535fb67 ({username='slave'} at 127.0.0.1:40034)
I20240503 14:01:34.405452 4170 tablet_copy_service.cc:161] P 6d05d524f254418bb351065633a4606c: Beginning new tablet copy session on tablet 11c78a8dc746409aa487dcd7ab58cc82 from peer e4f6fd3611bf4f0c812a03bde535fb67 at {username='slave'} at 127.0.0.1:40034: session id = e4f6fd3611bf4f0c812a03bde535fb67-11c78a8dc746409aa487dcd7ab58cc82
I20240503 14:01:34.406451 4170 tablet_copy_source_session.cc:205] T 11c78a8dc746409aa487dcd7ab58cc82 P 6d05d524f254418bb351065633a4606c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:34.407229 4780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c78a8dc746409aa487dcd7ab58cc82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:34.409658 4780 tablet_copy_client.cc:783] T 11c78a8dc746409aa487dcd7ab58cc82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:34.409819 4780 tablet_copy_client.cc:647] T 11c78a8dc746409aa487dcd7ab58cc82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:34.410812 4784 tablet_copy_client.cc:1002] T 11c78a8dc746409aa487dcd7ab58cc82 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:34.434144 4787 ts_tablet_manager.cc:902] T 8af6f6a1be8f43a693011cfa1c022f08 P 6d05d524f254418bb351065633a4606c: Initiating tablet copy from peer f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053)
I20240503 14:01:34.434829 4787 tablet_copy_client.cc:308] T 8af6f6a1be8f43a693011cfa1c022f08 P 6d05d524f254418bb351065633a4606c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:38053
I20240503 14:01:34.435225 3774 tablet_copy_service.cc:140] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Received BeginTabletCopySession request for tablet 8af6f6a1be8f43a693011cfa1c022f08 from peer 6d05d524f254418bb351065633a4606c ({username='slave'} at 127.0.0.1:33898)
I20240503 14:01:34.435328 3774 tablet_copy_service.cc:161] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Beginning new tablet copy session on tablet 8af6f6a1be8f43a693011cfa1c022f08 from peer 6d05d524f254418bb351065633a4606c at {username='slave'} at 127.0.0.1:33898: session id = 6d05d524f254418bb351065633a4606c-8af6f6a1be8f43a693011cfa1c022f08
I20240503 14:01:34.436368 3774 tablet_copy_source_session.cc:205] T 8af6f6a1be8f43a693011cfa1c022f08 P f71f8fdeb4004bb8b8012dc8b79ff8e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:34.437314 4787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af6f6a1be8f43a693011cfa1c022f08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:34.439841 4787 tablet_copy_client.cc:783] T 8af6f6a1be8f43a693011cfa1c022f08 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:34.440001 4787 tablet_copy_client.cc:647] T 8af6f6a1be8f43a693011cfa1c022f08 P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:34.441053 4791 tablet_copy_client.cc:1002] T 8af6f6a1be8f43a693011cfa1c022f08 P 6d05d524f254418bb351065633a4606c: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:34.470626 3715 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240503 14:01:34.472432 3914 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8d6ad905b31541ffa6f2007d886b7643"
dest_uuid: "f6d3c1f17e6b4f42a813bef57d94fe87"
mode: GRACEFUL
new_leader_uuid: "2d8cd9c1e8e2469888742770004dcb11"
from {username='slave'} at 127.0.0.1:47982
I20240503 14:01:34.472584 3914 raft_consensus.cc:595] T 8d6ad905b31541ffa6f2007d886b7643 P f6d3c1f17e6b4f42a813bef57d94fe87 [term 5 LEADER]: Received request to transfer leadership to TS 2d8cd9c1e8e2469888742770004dcb11
I20240503 14:01:34.472868 3715 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240503 14:01:34.472977 3715 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240503 14:01:34.481930 4792 ts_tablet_manager.cc:902] T 11893d5b97174d7389005a373091703b P 6d05d524f254418bb351065633a4606c: Initiating tablet copy from peer f71f8fdeb4004bb8b8012dc8b79ff8e1 (127.0.99.1:38053)
I20240503 14:01:34.482522 4792 tablet_copy_client.cc:308] T 11893d5b97174d7389005a373091703b P 6d05d524f254418bb351065633a4606c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.1:38053
I20240503 14:01:34.482910 3774 tablet_copy_service.cc:140] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Received BeginTabletCopySession request for tablet 11893d5b97174d7389005a373091703b from peer 6d05d524f254418bb351065633a4606c ({username='slave'} at 127.0.0.1:33898)
I20240503 14:01:34.483021 3774 tablet_copy_service.cc:161] P f71f8fdeb4004bb8b8012dc8b79ff8e1: Beginning new tablet copy session on tablet 11893d5b97174d7389005a373091703b from peer 6d05d524f254418bb351065633a4606c at {username='slave'} at 127.0.0.1:33898: session id = 6d05d524f254418bb351065633a4606c-11893d5b97174d7389005a373091703b
I20240503 14:01:34.484023 3774 tablet_copy_source_session.cc:205] T 11893d5b97174d7389005a373091703b P f71f8fdeb4004bb8b8012dc8b79ff8e1: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:34.484798 4792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11893d5b97174d7389005a373091703b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:34.487171 4792 tablet_copy_client.cc:783] T 11893d5b97174d7389005a373091703b P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:34.487329 4792 tablet_copy_client.cc:647] T 11893d5b97174d7389005a373091703b P 6d05d524f254418bb351065633a4606c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:34.488286 4796 tablet_copy_client.cc:1002] T 11893d5b97174d7389005a373091703b P 6d05d524f254418bb351065633a4606c: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:34.517747 4797 ts_tablet_manager.cc:902] T 6b7539be4b0c41d1accce62f9abbc9e9 P e4f6fd3611bf4f0c812a03bde535fb67: Initiating tablet copy from peer 06740b673b674aecadcbb0fa2bcac68e (127.0.99.6:33057)
I20240503 14:01:34.518280 4797 tablet_copy_client.cc:308] T 6b7539be4b0c41d1accce62f9abbc9e9 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Beginning tablet copy session from remote peer at address 127.0.99.6:33057
I20240503 14:01:34.518702 4321 tablet_copy_service.cc:140] P 06740b673b674aecadcbb0fa2bcac68e: Received BeginTabletCopySession request for tablet 6b7539be4b0c41d1accce62f9abbc9e9 from peer e4f6fd3611bf4f0c812a03bde535fb67 ({username='slave'} at 127.0.0.1:59054)
I20240503 14:01:34.518790 4321 tablet_copy_service.cc:161] P 06740b673b674aecadcbb0fa2bcac68e: Beginning new tablet copy session on tablet 6b7539be4b0c41d1accce62f9abbc9e9 from peer e4f6fd3611bf4f0c812a03bde535fb67 at {username='slave'} at 127.0.0.1:59054: session id = e4f6fd3611bf4f0c812a03bde535fb67-6b7539be4b0c41d1accce62f9abbc9e9
I20240503 14:01:34.519867 4321 tablet_copy_source_session.cc:205] T 6b7539be4b0c41d1accce62f9abbc9e9 P 06740b673b674aecadcbb0fa2bcac68e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:01:34.520684 4797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b7539be4b0c41d1accce62f9abbc9e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:01:34.523046 4797 tablet_copy_client.cc:783] T 6b7539be4b0c41d1accce62f9abbc9e9 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 0 data blocks...
I20240503 14:01:34.523234 4797 tablet_copy_client.cc:647] T 6b7539be4b0c41d1accce62f9abbc9e9 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Starting download of 1 WAL segments...
I20240503 14:01:34.524360 4801 tablet_copy_client.cc:1002] T 6b7539be4b0c41d1accce62f9abbc9e9 P e4f6fd3611bf4f0c812a03bde535fb67: tablet copy: Injecting latency into file download: 2000
I20240503 14:01:34.541186 4802 ts_tablet_manager.cc:902] T cc6eba6dbb2c4efda6d24f9ee9d6e8f7 P 06740b673b674aecadcbb0fa2bcac68e: Initiating tablet copy from peer f6d3c1f17e6b4f42a813bef57d94fe87 (127.0.99.3:43425)
F20240503 14:01:34.541447 4802 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0171f05f97a24f97ab8732925b360c82.queue_time_us, tablet-download-pool-0171f05f97a24f97ab8732925b360c82.queue_time_us) (tablet-download-pool-0171f05f97a24f97ab8732925b360c82.run_cpu_time_us, tablet-download-pool-0171f05f97a24f97ab8732925b360c82.run_cpu_time_us) (tablet-download-pool-0171f05f97a24f97ab8732925b360c82.run_wall_time_us, tablet-download-pool-0171f05f97a24f97ab8732925b360c82.run_wall_time_us) (tablet-download-pool-11893d5b97174d7389005a373091703b.queue_time_us, tablet-download-pool-11893d5b97174d7389005a373091703b.queue_time_us) (tablet-download-pool-11893d5b97174d7389005a373091703b.run_cpu_time_us, tablet-download-pool-11893d5b97174d7389005a373091703b.run_cpu_time_us) (tablet-download-pool-11893d5b97174d7389005a373091703b.run_wall_time_us, tablet-download-pool-11893d5b97174d7389005a373091703b.run_wall_time_us) (tablet-download-pool-11c78a8dc746409aa487dcd7ab58cc82.queue_time_us, tablet-download-pool-11c78a8dc746409aa487dcd7ab58cc82.queue_time_us) (tablet-download-pool-11c78a8dc746409aa487dcd7ab58cc82.run_cpu_time_us, tablet-download-pool-11c78a8dc746409aa487dcd7ab58cc82.run_cpu_time_us) (tablet-download-pool-11c78a8dc746409aa487dcd7ab58cc82.run_wall_time_us, tablet-download-pool-11c78a8dc746409aa487dcd7ab58cc82.run_wall_time_us) (tablet-download-pool-6b7539be4b0c41d1accce62f9abbc9e9.queue_time_us, tablet-download-pool-6b7539be4b0c41d1accce62f9abbc9e9.queue_time_us) (tablet-download-pool-6b7539be4b0c41d1accce62f9abbc9e9.run_cpu_time_us, tablet-download-pool-6b7539be4b0c41d1accce62f9abbc9e9.run_cpu_time_us) (tablet-download-pool-6b7539be4b0c41d1accce62f9abbc9e9.run_wall_time_us, tablet-download-pool-6b7539be4b0c41d1accce62f9abbc9e9.run_wall_time_us) (tablet-download-pool-71dd0142015e4fb791413d533b9ccb55.queue_time_us, tablet-download-pool-71dd0142015e4fb791413d533b9ccb55.queue_time_us) (tablet-download-pool-71dd0142015e4fb791413d533b9ccb55.run_cpu_time_us, tablet-download-pool-71dd0142015e4fb791413d533b9ccb55.run_cpu_time_us) (tablet-download-pool-71dd0142015e4fb791413d533b9ccb55.run_wall_time_us, tablet-download-pool-71dd0142015e4fb791413d533b9ccb55.run_wall_time_us) (tablet-download-pool-73cc3fa988034bb08f21b1613ad25b2e.queue_time_us, tablet-download-pool-73cc3fa988034bb08f21b1613ad25b2e.queue_time_us) (tablet-download-pool-73cc3fa988034bb08f21b1613ad25b2e.run_cpu_time_us, tablet-download-pool-73cc3fa988034bb08f21b1613ad25b2e.run_cpu_time_us) (tablet-download-pool-73cc3fa988034bb08f21b1613ad25b2e.run_wall_time_us, tablet-download-pool-73cc3fa988034bb08f21b1613ad25b2e.run_wall_time_us) (tablet-download-pool-8af6f6a1be8f43a693011cfa1c022f08.queue_time_us, tablet-download-pool-8af6f6a1be8f43a693011cfa1c022f08.queue_time_us) (tablet-download-pool-8af6f6a1be8f43a693011cfa1c022f08.run_cpu_time_us, tablet-download-pool-8af6f6a1be8f43a693011cfa1c022f08.run_cpu_time_us) (tablet-download-pool-8af6f6a1be8f43a693011cfa1c022f08.run_wall_time_us, tablet-download-pool-8af6f6a1be8f43a693011cfa1c022f08.run_wall_time_us) (tablet-download-pool-8d6ad905b31541ffa6f2007d886b7643.queue_time_us, tablet-download-pool-8d6ad905b31541ffa6f2007d886b7643.queue_time_us) (tablet-download-pool-8d6ad905b31541ffa6f2007d886b7643.run_cpu_time_us, tablet-download-pool-8d6ad905b31541ffa6f2007d886b7643.run_cpu_time_us) (tablet-download-pool-8d6ad905b31541ffa6f2007d886b7643.run_wall_time_us, tablet-download-pool-8d6ad905b31541ffa6f2007d886b7643.run_wall_time_us) (tablet-download-pool-95a2d80baec84cefb6d8b5e0ca27e5d7.queue_time_us, tablet-download-pool-95a2d80baec84cefb6d8b5e0ca27e5d7.queue_time_us) (tablet-download-pool-95a2d80baec84cefb6d8b5e0ca27e5d7.run_cpu_time_us, tablet-download-pool-95a2d80baec84cefb6d8b5e0ca27e5d7.run_cpu_time_us) (tablet-download-pool-95a2d80baec84cefb6d8b5e0ca27e5d7.run_wall_time_us, tablet-download-pool-95a2d80baec84cefb6d8b5e0ca27e5d7.run_wall_time_us) (tablet-download-pool-95d4c1bc93314dbea0dc3a714c8c9ec6.queue_time_us, tablet-download-pool-95d4c1bc93314dbea0dc3a714c8c9ec6.queue_time_us) (tablet-download-pool-95d4c1bc93314dbea0dc3a714c8c9ec6.run_cpu_time_us, tablet-download-pool-95d4c1bc93314dbea0dc3a714c8c9ec6.run_cpu_time_us) (tablet-download-pool-95d4c1bc93314dbea0dc3a714c8c9ec6.run_wall_time_us, tablet-download-pool-95d4c1bc93314dbea0dc3a714c8c9ec6.run_wall_time_us) (tablet-download-pool-b1bde81d4ddf4de3a2e16be46d97b1b5.queue_time_us, tablet-download-pool-b1bde81d4ddf4de3a2e16be46d97b1b5.queue_time_us) (tablet-download-pool-b1bde81d4ddf4de3a2e16be46d97b1b5.run_cpu_time_us, tablet-download-pool-b1bde81d4ddf4de3a2e16be46d97b1b5.run_cpu_time_us) (tablet-download-pool-b1bde81d4ddf4de3a2e16be46d97b1b5.run_wall_time_us, tablet-download-pool-b1bde81d4ddf4de3a2e16be46d97b1b5.run_wall_time_us) (tablet-download-pool-cc6eba6dbb2c4efda6d24f9ee9d6e8f7.queue_time_us, tablet-download-pool-cc6eba6dbb2c4efda6d24f9ee9d6e8f7.queue_time_us) (tablet-download-pool-d121433c26cb4d89876ce434ba669006.queue_time_us, tablet-download-pool-d121433c26cb4d89876ce434ba669006.queue_time_us) (tablet-download-pool-d121433c26cb4d89876ce434ba669006.run_cpu_time_us, tablet-download-pool-d121433c26cb4d89876ce434ba669006.run_cpu_time_us) (tablet-download-pool-d121433c26cb4d89876ce434ba669006.run_wall_time_us, tablet-download-pool-d121433c26cb4d89876ce434ba669006.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714744894 (unix time) try "date -d @1714744894" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000018c) received by PID 396 (TID 0x7fe3b3493700) from PID 396; stack trace: ***
@ 0x7fe45291c040 (unknown) at ??:0
@ 0x7fe45291bfb7 gsignal at ??:0
@ 0x7fe45291d921 abort at ??:0
@ 0x7fe453620dcd google::LogMessage::Fail() at ??:0
@ 0x7fe453624b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe4536207cc google::LogMessage::Flush() at ??:0
@ 0x7fe453621f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe454178044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fe45415cf98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fe45415aefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fe4550d777a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fe4550e0a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fe45516176f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fe4551754f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe455160226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fe45516fc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe4556ca3cc std::function<>::operator()() at ??:0
@ 0x7fe45415fe06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe4541606ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe45416203b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe4556ca3cc std::function<>::operator()() at ??:0
@ 0x7fe45414f807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe4509516db start_thread at ??:0
@ 0x7fe4529fe71f clone at ??:0