[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 19:55:57.011670 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:45753,127.25.83.61:37043,127.25.83.60:35639
I20240501 19:55:57.012650 25932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:57.013191 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.020246 25940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.020907 25938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.021807 25937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.022542 25932 server_base.cc:1034] running on GCE node
I20240501 19:55:57.023000 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:57.023094 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:57.023124 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593357023123 us; error 0 us; skew 500 ppm
I20240501 19:55:57.024801 25932 webserver.cc:460] Webserver started at http://127.25.83.62:43625/ using document root <none> and password file <none>
I20240501 19:55:57.025274 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.025322 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.025492 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.027125 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "ff0d01b59d9c46a28df732abff9fdda3"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.030367 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:57.032284 25945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.033265 25932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.033352 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "ff0d01b59d9c46a28df732abff9fdda3"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.033439 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.056847 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.057591 25932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:57.057755 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.065025 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:45753
I20240501 19:55:57.065078 25997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:45753 every 8 connection(s)
I20240501 19:55:57.066244 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:55:57.068531 25998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 19:55:57.068671 26000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.069317 26001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.069964 26003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.070967 25932 server_base.cc:1034] running on GCE node
I20240501 19:55:57.071156 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:57.071208 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:57.071229 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593357071229 us; error 0 us; skew 500 ppm
I20240501 19:55:57.072278 25932 webserver.cc:460] Webserver started at http://127.25.83.61:45001/ using document root <none> and password file <none>
I20240501 19:55:57.072435 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.072492 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.072562 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.072916 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root/instance:
uuid: "8fa13cfec0ec4b58b5206d9a651a97d1"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.073550 25998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:55:57.074939 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.076016 26008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.076290 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:55:57.076352 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root
uuid: "8fa13cfec0ec4b58b5206d9a651a97d1"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.076414 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.083848 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.084082 25998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:55:57.084239 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240501 19:55:57.084750 25946 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.83.61:37043: connect: Connection refused (error 111)
W20240501 19:55:57.085496 25998 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.83.61:37043: Network error: Client connection negotiation failed: client connection to 127.25.83.61:37043: connect: Connection refused (error 111)
I20240501 19:55:57.088918 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.61:37043
I20240501 19:55:57.088966 26063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.61:37043 every 8 connection(s)
I20240501 19:55:57.089519 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:55:57.089702 26064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:57.090878 26064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:55:57.091849 26066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.092592 26067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.093372 25932 server_base.cc:1034] running on GCE node
W20240501 19:55:57.094089 26069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.094399 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:57.094458 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:57.094480 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593357094480 us; error 0 us; skew 500 ppm
I20240501 19:55:57.095023 26064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:55:57.095418 25932 webserver.cc:460] Webserver started at http://127.25.83.60:45331/ using document root <none> and password file <none>
I20240501 19:55:57.095563 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.095655 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.095732 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.096133 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root/instance:
uuid: "dabccb401baa49b29fb783abb3121818"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.097682 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:55:57.098721 26076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.098950 26064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:55:57.098999 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 19:55:57.099048 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root
uuid: "dabccb401baa49b29fb783abb3121818"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.099104 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240501 19:55:57.100068 26064 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.83.60:35639: Network error: Client connection negotiation failed: client connection to 127.25.83.60:35639: connect: Connection refused (error 111)
I20240501 19:55:57.128118 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.128526 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.132944 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.60:35639
I20240501 19:55:57.132993 26129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.60:35639 every 8 connection(s)
I20240501 19:55:57.133340 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:55:57.133738 26130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:57.134655 25998 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } attempt: 1
I20240501 19:55:57.134727 26130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:55:57.138702 26130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:55:57.139104 25998 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:55:57.142545 26130 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:55:57.145280 25998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3: Bootstrap starting.
I20240501 19:55:57.146875 26130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: Bootstrap starting.
I20240501 19:55:57.148144 26130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:57.148607 25998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:57.149389 26130 log.cc:824] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:57.151244 26130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: No bootstrap required, opened a new log
I20240501 19:55:57.151353 25998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3: No bootstrap required, opened a new log
I20240501 19:55:57.152276 26064 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } attempt: 1
I20240501 19:55:57.155596 25998 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.155612 26130 raft_consensus.cc:348] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.155833 25998 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:57.155848 26130 raft_consensus.cc:374] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:57.155884 25998 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0d01b59d9c46a28df732abff9fdda3, State: Initialized, Role: FOLLOWER
I20240501 19:55:57.155884 26130 raft_consensus.cc:729] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dabccb401baa49b29fb783abb3121818, State: Initialized, Role: FOLLOWER
I20240501 19:55:57.156389 25998 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.156389 26130 consensus_queue.cc:260] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.156888 26136 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.157099 26136 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:55:57.157318 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.157456 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:55:57.157675 25998 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:57.157675 26130 sys_catalog.cc:564] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:57.157826 26064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1: Bootstrap starting.
I20240501 19:55:57.158737 26064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:57.159999 26064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1: No bootstrap required, opened a new log
I20240501 19:55:57.160425 26064 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.160506 26064 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:57.160532 26064 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fa13cfec0ec4b58b5206d9a651a97d1, State: Initialized, Role: FOLLOWER
I20240501 19:55:57.160650 26064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.160965 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.161048 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:55:57.161082 26064 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: configured and running, proceeding with master startup.
W20240501 19:55:57.162986 26160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:55:57.163064 26160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 19:55:57.162983 26159 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:55:57.163141 26159 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:55:57.163277 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240501 19:55:57.163631 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 19:55:57.164294 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.164439 26173 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:55:57.164510 26173 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 19:55:57.167317 26174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.167879 26175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.168663 26177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.168982 26161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:55:57.169127 26161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.169379 25932 server_base.cc:1034] running on GCE node
I20240501 19:55:57.169565 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:57.169611 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:57.169634 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593357169633 us; error 0 us; skew 500 ppm
I20240501 19:55:57.170053 26161 leader_election.cc:288] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff0d01b59d9c46a28df732abff9fdda3 (127.25.83.62:45753), dabccb401baa49b29fb783abb3121818 (127.25.83.60:35639)
I20240501 19:55:57.170675 25972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff0d01b59d9c46a28df732abff9fdda3" is_pre_election: true
I20240501 19:55:57.170881 25972 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa13cfec0ec4b58b5206d9a651a97d1 in term 0.
I20240501 19:55:57.171010 25932 webserver.cc:460] Webserver started at http://127.25.83.1:37447/ using document root <none> and password file <none>
I20240501 19:55:57.171161 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.171221 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.171188 26104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabccb401baa49b29fb783abb3121818" is_pre_election: true
I20240501 19:55:57.171283 26104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8fa13cfec0ec4b58b5206d9a651a97d1 in term 0.
I20240501 19:55:57.171288 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.171310 26011 leader_election.cc:302] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fa13cfec0ec4b58b5206d9a651a97d1, ff0d01b59d9c46a28df732abff9fdda3; no voters:
I20240501 19:55:57.171691 26161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:55:57.171752 26161 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:55:57.171792 26161 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:57.171799 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root/instance:
uuid: "b9fa8977446f4dc49fe26e95e895ad1d"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.173122 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:55:57.173189 26161 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.173511 26161 leader_election.cc:288] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [CANDIDATE]: Term 1 election: Requested vote from peers ff0d01b59d9c46a28df732abff9fdda3 (127.25.83.62:45753), dabccb401baa49b29fb783abb3121818 (127.25.83.60:35639)
I20240501 19:55:57.173841 26104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dabccb401baa49b29fb783abb3121818"
I20240501 19:55:57.173924 26104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:57.174216 25972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff0d01b59d9c46a28df732abff9fdda3"
I20240501 19:55:57.174310 25972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:57.174885 26104 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa13cfec0ec4b58b5206d9a651a97d1 in term 1.
I20240501 19:55:57.175169 26012 leader_election.cc:302] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8fa13cfec0ec4b58b5206d9a651a97d1, dabccb401baa49b29fb783abb3121818; no voters:
I20240501 19:55:57.175197 25972 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8fa13cfec0ec4b58b5206d9a651a97d1 in term 1.
I20240501 19:55:57.175487 26161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:57.175657 26161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 1 LEADER]: Becoming Leader. State: Replica: 8fa13cfec0ec4b58b5206d9a651a97d1, State: Running, Role: LEADER
I20240501 19:55:57.175716 26182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.175954 25932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:57.175952 26161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } }
I20240501 19:55:57.176043 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root
uuid: "b9fa8977446f4dc49fe26e95e895ad1d"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.176124 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.177644 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fa13cfec0ec4b58b5206d9a651a97d1. Latest consensus state: current_term: 1 leader_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.177745 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:57.178092 26189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:57.179049 26189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:57.183697 25972 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 1 FOLLOWER]: Refusing update from remote peer 8fa13cfec0ec4b58b5206d9a651a97d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:55:57.183697 26104 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 1 FOLLOWER]: Refusing update from remote peer 8fa13cfec0ec4b58b5206d9a651a97d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:55:57.184299 26183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:55:57.184434 26161 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:55:57.185997 26136 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fa13cfec0ec4b58b5206d9a651a97d1. Latest consensus state: current_term: 1 leader_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.186134 26136 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:55:57.185999 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fa13cfec0ec4b58b5206d9a651a97d1. Latest consensus state: current_term: 1 leader_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.186292 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:55:57.187462 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.187543 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:57.188308 26137 sys_catalog.cc:455] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.188410 26137 sys_catalog.cc:458] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:55:57.189049 26189 catalog_manager.cc:1347] Generated new cluster ID: 33957a26691e40d9b5861615c6d6ff7a
I20240501 19:55:57.189129 26189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:57.191316 26196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.191414 26196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:57.191987 26136 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff0d01b59d9c46a28df732abff9fdda3" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45753 } } peers { permanent_uuid: "8fa13cfec0ec4b58b5206d9a651a97d1" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 37043 } } peers { permanent_uuid: "dabccb401baa49b29fb783abb3121818" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 35639 } } }
I20240501 19:55:57.192076 26136 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:55:57.200094 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.200503 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.200894 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.201702 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.201756 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.201805 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.201833 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.207854 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.1:46511
I20240501 19:55:57.207959 26257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.1:46511 every 8 connection(s)
I20240501 19:55:57.209739 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:55:57.211272 26189 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:57.212276 26189 catalog_manager.cc:1504] Loading token signing keys...
W20240501 19:55:57.218503 26263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.218957 26265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.221467 26262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.222858 26259 heartbeater.cc:347] Connected to a master server at 127.25.83.62:45753
I20240501 19:55:57.223093 26259 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.223614 26259 heartbeater.cc:511] Master 127.25.83.62:45753 requested a full tablet report, sending...
I20240501 19:55:57.224016 26189 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1: Generated new TSK 0
I20240501 19:55:57.224722 26189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:57.224910 26258 heartbeater.cc:347] Connected to a master server at 127.25.83.60:35639
I20240501 19:55:57.224977 26258 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.225052 25962 ts_manager.cc:194] Registered new tserver with Master: b9fa8977446f4dc49fe26e95e895ad1d (127.25.83.1:46511)
I20240501 19:55:57.225150 26258 heartbeater.cc:511] Master 127.25.83.60:35639 requested a full tablet report, sending...
I20240501 19:55:57.225476 25932 server_base.cc:1034] running on GCE node
I20240501 19:55:57.225713 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 19:55:57.225754 26094 ts_manager.cc:194] Registered new tserver with Master: b9fa8977446f4dc49fe26e95e895ad1d (127.25.83.1:46511)
W20240501 19:55:57.225761 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:57.225975 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593357225974 us; error 0 us; skew 500 ppm
I20240501 19:55:57.226914 26260 heartbeater.cc:347] Connected to a master server at 127.25.83.61:37043
I20240501 19:55:57.226939 25932 webserver.cc:460] Webserver started at http://127.25.83.2:42801/ using document root <none> and password file <none>
I20240501 19:55:57.227017 26260 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.227109 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.227162 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.227212 26260 heartbeater.cc:511] Master 127.25.83.61:37043 requested a full tablet report, sending...
I20240501 19:55:57.227237 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.227686 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root/instance:
uuid: "9cc73b95a7c148199c7fc42ff2ed2a07"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.227731 26028 ts_manager.cc:194] Registered new tserver with Master: b9fa8977446f4dc49fe26e95e895ad1d (127.25.83.1:46511)
I20240501 19:55:57.228701 26028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60330
I20240501 19:55:57.229382 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:55:57.230335 26273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.230607 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:55:57.230667 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root
uuid: "9cc73b95a7c148199c7fc42ff2ed2a07"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.230724 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.237438 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.237807 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.238080 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.238548 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.238593 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.238643 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.238679 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.243127 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.2:45067
I20240501 19:55:57.243239 26336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.2:45067 every 8 connection(s)
I20240501 19:55:57.244060 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:57.246873 26344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:57.247567 26345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.248271 25932 server_base.cc:1034] running on GCE node
W20240501 19:55:57.248965 26347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:57.249158 26337 heartbeater.cc:347] Connected to a master server at 127.25.83.60:35639
I20240501 19:55:57.249251 26337 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.249464 26337 heartbeater.cc:511] Master 127.25.83.60:35639 requested a full tablet report, sending...
I20240501 19:55:57.249930 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:57.250335 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:57.250418 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593357250417 us; error 0 us; skew 500 ppm
I20240501 19:55:57.250254 26094 ts_manager.cc:194] Registered new tserver with Master: 9cc73b95a7c148199c7fc42ff2ed2a07 (127.25.83.2:45067)
I20240501 19:55:57.250909 26338 heartbeater.cc:347] Connected to a master server at 127.25.83.62:45753
I20240501 19:55:57.250977 26338 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.251159 26338 heartbeater.cc:511] Master 127.25.83.62:45753 requested a full tablet report, sending...
I20240501 19:55:57.251720 25932 webserver.cc:460] Webserver started at http://127.25.83.3:34639/ using document root <none> and password file <none>
I20240501 19:55:57.251718 25962 ts_manager.cc:194] Registered new tserver with Master: 9cc73b95a7c148199c7fc42ff2ed2a07 (127.25.83.2:45067)
I20240501 19:55:57.251973 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:57.252039 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:57.252117 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:57.252559 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root/instance:
uuid: "b2a450999d1e4b98a32e5466cd1006d0"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.252696 26339 heartbeater.cc:347] Connected to a master server at 127.25.83.61:37043
I20240501 19:55:57.252770 26339 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.252980 26339 heartbeater.cc:511] Master 127.25.83.61:37043 requested a full tablet report, sending...
I20240501 19:55:57.253536 26028 ts_manager.cc:194] Registered new tserver with Master: 9cc73b95a7c148199c7fc42ff2ed2a07 (127.25.83.2:45067)
I20240501 19:55:57.254240 26028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60336
I20240501 19:55:57.254535 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:57.255426 26352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.255641 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:55:57.255708 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root
uuid: "b2a450999d1e4b98a32e5466cd1006d0"
format_stamp: "Formatted at 2024-05-01 19:55:57 on dist-test-slave-mfp3"
I20240501 19:55:57.255772 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:57.277864 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:57.278411 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:57.278735 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:57.279294 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:57.279343 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.279387 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:57.279417 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:57.285283 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.3:35027
I20240501 19:55:57.285358 26415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.3:35027 every 8 connection(s)
I20240501 19:55:57.291715 26418 heartbeater.cc:347] Connected to a master server at 127.25.83.61:37043
I20240501 19:55:57.291831 26418 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.292076 26418 heartbeater.cc:511] Master 127.25.83.61:37043 requested a full tablet report, sending...
I20240501 19:55:57.292083 26416 heartbeater.cc:347] Connected to a master server at 127.25.83.60:35639
I20240501 19:55:57.292201 26416 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.292393 26416 heartbeater.cc:511] Master 127.25.83.60:35639 requested a full tablet report, sending...
I20240501 19:55:57.292416 26417 heartbeater.cc:347] Connected to a master server at 127.25.83.62:45753
I20240501 19:55:57.292485 26417 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:57.292646 26417 heartbeater.cc:511] Master 127.25.83.62:45753 requested a full tablet report, sending...
I20240501 19:55:57.292668 26028 ts_manager.cc:194] Registered new tserver with Master: b2a450999d1e4b98a32e5466cd1006d0 (127.25.83.3:35027)
I20240501 19:55:57.293246 25962 ts_manager.cc:194] Registered new tserver with Master: b2a450999d1e4b98a32e5466cd1006d0 (127.25.83.3:35027)
I20240501 19:55:57.293257 26028 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60342
I20240501 19:55:57.293761 26094 ts_manager.cc:194] Registered new tserver with Master: b2a450999d1e4b98a32e5466cd1006d0 (127.25.83.3:35027)
I20240501 19:55:57.293946 25932 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008104317s
I20240501 19:55:58.164400 26160 catalog_manager.cc:1259] Loaded cluster ID: 33957a26691e40d9b5861615c6d6ff7a
I20240501 19:55:58.164400 26159 catalog_manager.cc:1259] Loaded cluster ID: 33957a26691e40d9b5861615c6d6ff7a
I20240501 19:55:58.164626 26160 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3: loading cluster ID for follower catalog manager: success
I20240501 19:55:58.164637 26159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: loading cluster ID for follower catalog manager: success
I20240501 19:55:58.166519 26159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: acquiring CA information for follower catalog manager: success
I20240501 19:55:58.166519 26160 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3: acquiring CA information for follower catalog manager: success
I20240501 19:55:58.167325 26160 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:55:58.167325 26159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:55:58.229853 26260 heartbeater.cc:503] Master 127.25.83.61:37043 was elected leader, sending a full tablet report...
I20240501 19:55:58.255368 26339 heartbeater.cc:503] Master 127.25.83.61:37043 was elected leader, sending a full tablet report...
I20240501 19:55:58.294314 26418 heartbeater.cc:503] Master 127.25.83.61:37043 was elected leader, sending a full tablet report...
I20240501 19:55:59.163722 26172 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:55:59.348170 25932 test_util.cc:268] Using random seed: -534888002
I20240501 19:55:59.371722 26028 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43316:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:55:59.389302 26380 tablet_service.cc:1467] Processing CreateTablet for tablet e3ddaeb74d2c4c65a008be01c0f3cd49 (DEFAULT_TABLE table=test-workload [id=c80b498e2d874832a850e86a10076314]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20240501 19:55:59.389302 26221 tablet_service.cc:1467] Processing CreateTablet for tablet 75fd74b39ee34710a76a3b38052cb8b2 (DEFAULT_TABLE table=test-workload [id=c80b498e2d874832a850e86a10076314]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20240501 19:55:59.389302 26222 tablet_service.cc:1467] Processing CreateTablet for tablet bc4645b951c2429d9ab7d398c8c77352 (DEFAULT_TABLE table=test-workload [id=c80b498e2d874832a850e86a10076314]), partition=RANGE (key) PARTITION VALUES < 536870911
I20240501 19:55:59.389752 26380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3ddaeb74d2c4c65a008be01c0f3cd49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:59.389756 26221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75fd74b39ee34710a76a3b38052cb8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:59.389958 26222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4645b951c2429d9ab7d398c8c77352. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:59.392180 26452 tablet_bootstrap.cc:492] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d: Bootstrap starting.
I20240501 19:55:59.392401 26301 tablet_service.cc:1467] Processing CreateTablet for tablet 9bfff83f79c14d428e97d98ece591e2f (DEFAULT_TABLE table=test-workload [id=c80b498e2d874832a850e86a10076314]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20240501 19:55:59.392643 26301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bfff83f79c14d428e97d98ece591e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:59.393067 26452 tablet_bootstrap.cc:654] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:59.393612 26451 tablet_bootstrap.cc:492] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0: Bootstrap starting.
I20240501 19:55:59.394080 26452 tablet_bootstrap.cc:492] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d: No bootstrap required, opened a new log
I20240501 19:55:59.394183 26452 ts_tablet_manager.cc:1367] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:59.394426 26454 tablet_bootstrap.cc:492] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07: Bootstrap starting.
I20240501 19:55:59.394522 26452 raft_consensus.cc:348] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } }
I20240501 19:55:59.394611 26452 raft_consensus.cc:374] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:59.394649 26452 raft_consensus.cc:729] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fa8977446f4dc49fe26e95e895ad1d, State: Initialized, Role: FOLLOWER
I20240501 19:55:59.394757 26452 consensus_queue.cc:260] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } }
I20240501 19:55:59.394850 26452 raft_consensus.cc:388] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:59.394891 26452 raft_consensus.cc:482] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:59.394942 26452 raft_consensus.cc:3037] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:59.395304 26454 tablet_bootstrap.cc:654] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:59.395819 26452 raft_consensus.cc:504] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } }
I20240501 19:55:59.395957 26452 leader_election.cc:302] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9fa8977446f4dc49fe26e95e895ad1d; no voters:
I20240501 19:55:59.396137 26452 leader_election.cc:288] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:59.396270 26456 raft_consensus.cc:2781] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:59.396461 26456 raft_consensus.cc:686] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 LEADER]: Becoming Leader. State: Replica: b9fa8977446f4dc49fe26e95e895ad1d, State: Running, Role: LEADER
I20240501 19:55:59.396639 26454 tablet_bootstrap.cc:492] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07: No bootstrap required, opened a new log
I20240501 19:55:59.396584 26456 consensus_queue.cc:237] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } }
I20240501 19:55:59.396730 26454 ts_tablet_manager.cc:1367] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:59.396847 26451 tablet_bootstrap.cc:654] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:59.397069 26454 raft_consensus.cc:348] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc73b95a7c148199c7fc42ff2ed2a07" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 45067 } }
I20240501 19:55:59.397130 26454 raft_consensus.cc:374] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:59.397150 26454 raft_consensus.cc:729] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc73b95a7c148199c7fc42ff2ed2a07, State: Initialized, Role: FOLLOWER
I20240501 19:55:59.397243 26454 consensus_queue.cc:260] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc73b95a7c148199c7fc42ff2ed2a07" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 45067 } }
I20240501 19:55:59.397300 26454 raft_consensus.cc:388] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:59.397321 26454 raft_consensus.cc:482] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:59.397346 26454 raft_consensus.cc:3037] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:59.397375 26452 ts_tablet_manager.cc:1398] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:55:59.398063 26454 raft_consensus.cc:504] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc73b95a7c148199c7fc42ff2ed2a07" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 45067 } }
I20240501 19:55:59.398193 26454 leader_election.cc:302] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc73b95a7c148199c7fc42ff2ed2a07; no voters:
I20240501 19:55:59.398366 26454 leader_election.cc:288] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:59.398448 26460 raft_consensus.cc:2781] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:59.398593 26460 raft_consensus.cc:686] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 1 LEADER]: Becoming Leader. State: Replica: 9cc73b95a7c148199c7fc42ff2ed2a07, State: Running, Role: LEADER
I20240501 19:55:59.398661 26454 ts_tablet_manager.cc:1398] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.398698 26460 consensus_queue.cc:237] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc73b95a7c148199c7fc42ff2ed2a07" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 45067 } }
I20240501 19:55:59.398842 26028 catalog_manager.cc:5484] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fa8977446f4dc49fe26e95e895ad1d (127.25.83.1). New cstate: current_term: 1 leader_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } health_report { overall_health: HEALTHY } } }
I20240501 19:55:59.400291 26027 catalog_manager.cc:5484] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cc73b95a7c148199c7fc42ff2ed2a07 (127.25.83.2). New cstate: current_term: 1 leader_uuid: "9cc73b95a7c148199c7fc42ff2ed2a07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc73b95a7c148199c7fc42ff2ed2a07" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 45067 } health_report { overall_health: HEALTHY } } }
I20240501 19:55:59.401343 26452 tablet_bootstrap.cc:492] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d: Bootstrap starting.
I20240501 19:55:59.402021 26452 tablet_bootstrap.cc:654] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:59.403154 26451 tablet_bootstrap.cc:492] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0: No bootstrap required, opened a new log
I20240501 19:55:59.403158 26452 tablet_bootstrap.cc:492] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d: No bootstrap required, opened a new log
I20240501 19:55:59.403257 26451 ts_tablet_manager.cc:1367] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:55:59.403257 26452 ts_tablet_manager.cc:1367] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.403651 26451 raft_consensus.cc:348] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a450999d1e4b98a32e5466cd1006d0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35027 } }
I20240501 19:55:59.403729 26451 raft_consensus.cc:374] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:59.403762 26451 raft_consensus.cc:729] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a450999d1e4b98a32e5466cd1006d0, State: Initialized, Role: FOLLOWER
I20240501 19:55:59.403865 26451 consensus_queue.cc:260] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a450999d1e4b98a32e5466cd1006d0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35027 } }
I20240501 19:55:59.403967 26451 raft_consensus.cc:388] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:59.404013 26451 raft_consensus.cc:482] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:59.404073 26451 raft_consensus.cc:3037] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:59.404858 26451 raft_consensus.cc:504] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a450999d1e4b98a32e5466cd1006d0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35027 } }
I20240501 19:55:59.405004 26451 leader_election.cc:302] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2a450999d1e4b98a32e5466cd1006d0; no voters:
I20240501 19:55:59.405174 26451 leader_election.cc:288] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:59.405443 26466 raft_consensus.cc:2781] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:59.405658 26466 raft_consensus.cc:686] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 1 LEADER]: Becoming Leader. State: Replica: b2a450999d1e4b98a32e5466cd1006d0, State: Running, Role: LEADER
I20240501 19:55:59.405973 26466 consensus_queue.cc:237] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a450999d1e4b98a32e5466cd1006d0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35027 } }
I20240501 19:55:59.407027 26452 raft_consensus.cc:348] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } }
I20240501 19:55:59.407099 26452 raft_consensus.cc:374] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:59.407119 26452 raft_consensus.cc:729] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9fa8977446f4dc49fe26e95e895ad1d, State: Initialized, Role: FOLLOWER
I20240501 19:55:59.407202 26452 consensus_queue.cc:260] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } }
I20240501 19:55:59.407264 26452 raft_consensus.cc:388] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:59.407296 26452 raft_consensus.cc:482] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:59.407341 26452 raft_consensus.cc:3037] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:59.408071 26452 raft_consensus.cc:504] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } }
I20240501 19:55:59.408211 26452 leader_election.cc:302] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9fa8977446f4dc49fe26e95e895ad1d; no voters:
I20240501 19:55:59.408298 26452 leader_election.cc:288] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:59.408346 26456 raft_consensus.cc:2781] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:59.408501 26456 raft_consensus.cc:686] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 LEADER]: Becoming Leader. State: Replica: b9fa8977446f4dc49fe26e95e895ad1d, State: Running, Role: LEADER
I20240501 19:55:59.408638 26456 consensus_queue.cc:237] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } }
I20240501 19:55:59.408524 26452 ts_tablet_manager.cc:1398] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20240501 19:55:59.405473 26451 ts_tablet_manager.cc:1398] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 19:55:59.411553 26028 catalog_manager.cc:5484] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d reported cstate change: term changed from 0 to 1, leader changed from <none> to b9fa8977446f4dc49fe26e95e895ad1d (127.25.83.1). New cstate: current_term: 1 leader_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9fa8977446f4dc49fe26e95e895ad1d" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46511 } health_report { overall_health: HEALTHY } } }
I20240501 19:55:59.412233 26027 catalog_manager.cc:5484] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2a450999d1e4b98a32e5466cd1006d0 (127.25.83.3). New cstate: current_term: 1 leader_uuid: "b2a450999d1e4b98a32e5466cd1006d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a450999d1e4b98a32e5466cd1006d0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35027 } health_report { overall_health: HEALTHY } } }
I20240501 19:55:59.423032 25932 tablet_server.cc:178] TabletServer@127.25.83.1:0 shutting down...
I20240501 19:55:59.426738 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:55:59.426967 25932 tablet_replica.cc:330] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d: stopping tablet replica
I20240501 19:55:59.427181 25932 raft_consensus.cc:2229] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:55:59.427306 25932 raft_consensus.cc:2258] T bc4645b951c2429d9ab7d398c8c77352 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:55:59.428273 25932 tablet_replica.cc:330] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d: stopping tablet replica
I20240501 19:55:59.428371 25932 raft_consensus.cc:2229] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:55:59.428462 25932 raft_consensus.cc:2258] T 75fd74b39ee34710a76a3b38052cb8b2 P b9fa8977446f4dc49fe26e95e895ad1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:55:59.442034 25932 tablet_server.cc:194] TabletServer@127.25.83.1:0 shutdown complete.
I20240501 19:55:59.447371 25932 tablet_server.cc:178] TabletServer@127.25.83.2:0 shutting down...
I20240501 19:55:59.452350 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:55:59.452653 25932 tablet_replica.cc:330] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07: stopping tablet replica
I20240501 19:55:59.452754 25932 raft_consensus.cc:2229] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:55:59.452852 25932 raft_consensus.cc:2258] T 9bfff83f79c14d428e97d98ece591e2f P 9cc73b95a7c148199c7fc42ff2ed2a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:55:59.465754 25932 tablet_server.cc:194] TabletServer@127.25.83.2:0 shutdown complete.
I20240501 19:55:59.467582 25932 tablet_server.cc:178] TabletServer@127.25.83.3:0 shutting down...
I20240501 19:55:59.470739 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:55:59.470973 25932 tablet_replica.cc:330] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0: stopping tablet replica
I20240501 19:55:59.471076 25932 raft_consensus.cc:2229] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:55:59.471172 25932 raft_consensus.cc:2258] T e3ddaeb74d2c4c65a008be01c0f3cd49 P b2a450999d1e4b98a32e5466cd1006d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:55:59.483677 25932 tablet_server.cc:194] TabletServer@127.25.83.3:0 shutdown complete.
I20240501 19:55:59.485463 25932 master.cc:537] Master@127.25.83.62:45753 shutting down...
I20240501 19:55:59.488828 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:55:59.488963 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:55:59.489032 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P ff0d01b59d9c46a28df732abff9fdda3: stopping tablet replica
I20240501 19:55:59.510860 25932 master.cc:552] Master@127.25.83.62:45753 shutdown complete.
I20240501 19:55:59.513264 25932 master.cc:537] Master@127.25.83.61:37043 shutting down...
I20240501 19:55:59.516894 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:55:59.517129 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:55:59.517186 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8fa13cfec0ec4b58b5206d9a651a97d1: stopping tablet replica
I20240501 19:55:59.529752 25932 master.cc:552] Master@127.25.83.61:37043 shutdown complete.
I20240501 19:55:59.531965 25932 master.cc:537] Master@127.25.83.60:35639 shutting down...
I20240501 19:55:59.535414 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:55:59.535512 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:55:59.535552 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P dabccb401baa49b29fb783abb3121818: stopping tablet replica
I20240501 19:55:59.547698 25932 master.cc:552] Master@127.25.83.60:35639 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2544 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20240501 19:55:59.555116 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:44465
I20240501 19:55:59.555492 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.557855 26474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.558552 26477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.559146 26475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.560099 25932 server_base.cc:1034] running on GCE node
I20240501 19:55:59.560249 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:59.560288 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:59.560312 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593359560310 us; error 0 us; skew 500 ppm
I20240501 19:55:59.561246 25932 webserver.cc:460] Webserver started at http://127.25.83.62:37153/ using document root <none> and password file <none>
I20240501 19:55:59.561395 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.561447 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.561530 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.561909 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "22d7a02c52524429a61b4f8838e31adc"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-mfp3"
I20240501 19:55:59.563352 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:55:59.564219 26482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.564432 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:55:59.564496 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "22d7a02c52524429a61b4f8838e31adc"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-mfp3"
I20240501 19:55:59.564560 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.601001 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.601348 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.605089 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:44465
I20240501 19:55:59.608071 26534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:44465 every 8 connection(s)
I20240501 19:55:59.608536 26535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:59.610193 26535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc: Bootstrap starting.
I20240501 19:55:59.610878 26535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:59.611730 26535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc: No bootstrap required, opened a new log
I20240501 19:55:59.612043 26535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d7a02c52524429a61b4f8838e31adc" member_type: VOTER }
I20240501 19:55:59.612128 26535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:59.612174 26535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22d7a02c52524429a61b4f8838e31adc, State: Initialized, Role: FOLLOWER
I20240501 19:55:59.612295 26535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d7a02c52524429a61b4f8838e31adc" member_type: VOTER }
I20240501 19:55:59.612349 26535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:59.612381 26535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:59.612425 26535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:59.612987 26535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d7a02c52524429a61b4f8838e31adc" member_type: VOTER }
I20240501 19:55:59.613088 26535 leader_election.cc:302] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22d7a02c52524429a61b4f8838e31adc; no voters:
I20240501 19:55:59.613245 26535 leader_election.cc:288] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:59.613320 26538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:59.613566 26538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 1 LEADER]: Becoming Leader. State: Replica: 22d7a02c52524429a61b4f8838e31adc, State: Running, Role: LEADER
I20240501 19:55:59.613637 26535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:59.613688 26538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d7a02c52524429a61b4f8838e31adc" member_type: VOTER }
I20240501 19:55:59.614132 26540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22d7a02c52524429a61b4f8838e31adc. Latest consensus state: current_term: 1 leader_uuid: "22d7a02c52524429a61b4f8838e31adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d7a02c52524429a61b4f8838e31adc" member_type: VOTER } }
I20240501 19:55:59.614202 26540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:59.614068 26539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22d7a02c52524429a61b4f8838e31adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22d7a02c52524429a61b4f8838e31adc" member_type: VOTER } }
I20240501 19:55:59.614351 26539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:59.614717 26543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:59.615378 26543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:59.615514 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:55:59.616847 26543 catalog_manager.cc:1347] Generated new cluster ID: 686b69c453a04fbd877a2a37efa2229a
I20240501 19:55:59.616894 26543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:59.636292 26543 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:59.636816 26543 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:59.652069 26543 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc: Generated new TSK 0
I20240501 19:55:59.652287 26543 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:59.680032 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.682605 26557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.683518 26559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.684156 26556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.684772 25932 server_base.cc:1034] running on GCE node
I20240501 19:55:59.684906 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:59.684939 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:59.684954 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593359684954 us; error 0 us; skew 500 ppm
I20240501 19:55:59.685822 25932 webserver.cc:460] Webserver started at http://127.25.83.1:43077/ using document root <none> and password file <none>
I20240501 19:55:59.685946 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.685992 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.686050 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.686460 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-0-root/instance:
uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-mfp3"
I20240501 19:55:59.687839 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:55:59.688664 26564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.688881 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:55:59.688946 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-0-root
uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-mfp3"
I20240501 19:55:59.689009 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.694622 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.694931 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.695151 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.695523 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.695555 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.695592 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.695622 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.699033 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.1:39375
I20240501 19:55:59.699057 26627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.1:39375 every 8 connection(s)
I20240501 19:55:59.699739 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.701964 26632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.702560 26633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.703135 26635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.703491 26628 heartbeater.cc:347] Connected to a master server at 127.25.83.62:44465
I20240501 19:55:59.703562 26628 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.703719 26628 heartbeater.cc:511] Master 127.25.83.62:44465 requested a full tablet report, sending...
I20240501 19:55:59.704064 25932 server_base.cc:1034] running on GCE node
I20240501 19:55:59.704213 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:59.704253 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:59.704262 26499 ts_manager.cc:194] Registered new tserver with Master: df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375)
I20240501 19:55:59.704274 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593359704274 us; error 0 us; skew 500 ppm
I20240501 19:55:59.705066 25932 webserver.cc:460] Webserver started at http://127.25.83.2:42289/ using document root <none> and password file <none>
I20240501 19:55:59.705127 26499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58794
I20240501 19:55:59.705207 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.705257 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.705325 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.705673 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-1-root/instance:
uuid: "142ca6504e854a268686cc7bbc10936f"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-mfp3"
I20240501 19:55:59.707130 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:55:59.707883 26640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.708145 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:55:59.708223 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-1-root
uuid: "142ca6504e854a268686cc7bbc10936f"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-mfp3"
I20240501 19:55:59.708289 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.724767 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.725050 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.725266 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.725625 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.725656 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.725687 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.725714 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.729418 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.2:38931
I20240501 19:55:59.729486 26703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.2:38931 every 8 connection(s)
I20240501 19:55:59.730159 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.732347 26707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.732990 26708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.733561 26710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.733717 26704 heartbeater.cc:347] Connected to a master server at 127.25.83.62:44465
I20240501 19:55:59.733794 26704 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.733968 26704 heartbeater.cc:511] Master 127.25.83.62:44465 requested a full tablet report, sending...
I20240501 19:55:59.734486 25932 server_base.cc:1034] running on GCE node
I20240501 19:55:59.734498 26499 ts_manager.cc:194] Registered new tserver with Master: 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:55:59.734709 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:55:59.734761 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:55:59.734786 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593359734785 us; error 0 us; skew 500 ppm
I20240501 19:55:59.735128 26499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58806
I20240501 19:55:59.735735 25932 webserver.cc:460] Webserver started at http://127.25.83.3:38815/ using document root <none> and password file <none>
I20240501 19:55:59.735874 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.735922 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.736006 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.736341 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-2-root/instance:
uuid: "3d10550110544517b0590ca73d778ce4"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-mfp3"
I20240501 19:55:59.737694 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:55:59.738592 26715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.738839 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 19:55:59.738905 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-2-root
uuid: "3d10550110544517b0590ca73d778ce4"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-mfp3"
I20240501 19:55:59.738977 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.756372 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.756778 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.757107 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.757556 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.757596 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.757642 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.757678 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.762377 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.3:44891
I20240501 19:55:59.762415 26778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.3:44891 every 8 connection(s)
I20240501 19:55:59.767122 26779 heartbeater.cc:347] Connected to a master server at 127.25.83.62:44465
I20240501 19:55:59.767215 26779 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.767398 26779 heartbeater.cc:511] Master 127.25.83.62:44465 requested a full tablet report, sending...
I20240501 19:55:59.767997 26499 ts_manager.cc:194] Registered new tserver with Master: 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:55:59.768426 25932 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005634312s
I20240501 19:55:59.768520 26499 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58814
I20240501 19:56:00.706126 26628 heartbeater.cc:503] Master 127.25.83.62:44465 was elected leader, sending a full tablet report...
I20240501 19:56:00.736052 26704 heartbeater.cc:503] Master 127.25.83.62:44465 was elected leader, sending a full tablet report...
I20240501 19:56:00.769620 26779 heartbeater.cc:503] Master 127.25.83.62:44465 was elected leader, sending a full tablet report...
I20240501 19:56:00.794675 25932 test_util.cc:268] Using random seed: -533441489
I20240501 19:56:00.802644 26499 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58830:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:00.803181 26499 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:00.814090 26592 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6b7e7eb704466594924a582419499f (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 19:56:00.814440 26592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6b7e7eb704466594924a582419499f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.814496 26585 tablet_service.cc:1467] Processing CreateTablet for tablet 6afb91c7039d44fba7e4b3355421a01d (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 19:56:00.814754 26585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6afb91c7039d44fba7e4b3355421a01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.816066 26742 tablet_service.cc:1467] Processing CreateTablet for tablet 25fe56f01fff4c78a5b9759df9ada16e (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 19:56:00.816313 26742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fe56f01fff4c78a5b9759df9ada16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.817015 26586 tablet_service.cc:1467] Processing CreateTablet for tablet 9154e1e14b2d403fa4ab1fdbf2aaa972 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 19:56:00.817055 26743 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6b7e7eb704466594924a582419499f (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 19:56:00.817234 26586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154e1e14b2d403fa4ab1fdbf2aaa972. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.817317 26743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6b7e7eb704466594924a582419499f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.818428 26591 tablet_service.cc:1467] Processing CreateTablet for tablet 25fe56f01fff4c78a5b9759df9ada16e (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 19:56:00.818670 26591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fe56f01fff4c78a5b9759df9ada16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.819097 26587 tablet_service.cc:1467] Processing CreateTablet for tablet a72cd6084c9f4fbc9e881c09de7e4047 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 19:56:00.819238 26741 tablet_service.cc:1467] Processing CreateTablet for tablet d9f9a9d39e3a47f4b19e44cf9dcf8c34 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 19:56:00.819327 26587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72cd6084c9f4fbc9e881c09de7e4047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.819417 26741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f9a9d39e3a47f4b19e44cf9dcf8c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.820837 26740 tablet_service.cc:1467] Processing CreateTablet for tablet e603f27d210f4d4890043cddf48679ce (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 19:56:00.820998 26740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e603f27d210f4d4890043cddf48679ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.821074 26588 tablet_service.cc:1467] Processing CreateTablet for tablet 1e11c4a0fa824134a1cdd3caa6d494a7 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 19:56:00.821240 26588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e11c4a0fa824134a1cdd3caa6d494a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.822356 26590 tablet_service.cc:1467] Processing CreateTablet for tablet d9f9a9d39e3a47f4b19e44cf9dcf8c34 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 19:56:00.822402 26661 tablet_service.cc:1467] Processing CreateTablet for tablet 6afb91c7039d44fba7e4b3355421a01d (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 19:56:00.822582 26661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6afb91c7039d44fba7e4b3355421a01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.822584 26590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f9a9d39e3a47f4b19e44cf9dcf8c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.822824 26589 tablet_service.cc:1467] Processing CreateTablet for tablet e603f27d210f4d4890043cddf48679ce (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 19:56:00.823102 26589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e603f27d210f4d4890043cddf48679ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.823531 26739 tablet_service.cc:1467] Processing CreateTablet for tablet 1e11c4a0fa824134a1cdd3caa6d494a7 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 19:56:00.823737 26739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e11c4a0fa824134a1cdd3caa6d494a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.824615 26801 tablet_bootstrap.cc:492] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f: Bootstrap starting.
I20240501 19:56:00.825486 26801 tablet_bootstrap.cc:654] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.825733 26738 tablet_service.cc:1467] Processing CreateTablet for tablet a72cd6084c9f4fbc9e881c09de7e4047 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 19:56:00.825953 26738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72cd6084c9f4fbc9e881c09de7e4047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.826404 26662 tablet_service.cc:1467] Processing CreateTablet for tablet 9154e1e14b2d403fa4ab1fdbf2aaa972 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 19:56:00.826556 26801 tablet_bootstrap.cc:492] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f: No bootstrap required, opened a new log
I20240501 19:56:00.826614 26801 ts_tablet_manager.cc:1367] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:00.826618 26662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154e1e14b2d403fa4ab1fdbf2aaa972. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.826838 26663 tablet_service.cc:1467] Processing CreateTablet for tablet a72cd6084c9f4fbc9e881c09de7e4047 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20240501 19:56:00.827067 26663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a72cd6084c9f4fbc9e881c09de7e4047. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.827292 26664 tablet_service.cc:1467] Processing CreateTablet for tablet 1e11c4a0fa824134a1cdd3caa6d494a7 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20240501 19:56:00.827519 26664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e11c4a0fa824134a1cdd3caa6d494a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.827984 26737 tablet_service.cc:1467] Processing CreateTablet for tablet 9154e1e14b2d403fa4ab1fdbf2aaa972 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20240501 19:56:00.828241 26737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9154e1e14b2d403fa4ab1fdbf2aaa972. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.827061 26801 raft_consensus.cc:348] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.828970 26801 raft_consensus.cc:374] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.829000 26801 raft_consensus.cc:729] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.829111 26801 consensus_queue.cc:260] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.829439 26801 ts_tablet_manager.cc:1398] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:00.829588 26801 tablet_bootstrap.cc:492] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f: Bootstrap starting.
I20240501 19:56:00.830268 26736 tablet_service.cc:1467] Processing CreateTablet for tablet 6afb91c7039d44fba7e4b3355421a01d (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20240501 19:56:00.830374 26665 tablet_service.cc:1467] Processing CreateTablet for tablet e603f27d210f4d4890043cddf48679ce (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20240501 19:56:00.830487 26736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6afb91c7039d44fba7e4b3355421a01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.830567 26665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e603f27d210f4d4890043cddf48679ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.830615 26801 tablet_bootstrap.cc:654] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.831837 26801 tablet_bootstrap.cc:492] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f: No bootstrap required, opened a new log
I20240501 19:56:00.831928 26801 ts_tablet_manager.cc:1367] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:00.832271 26801 raft_consensus.cc:348] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.832365 26801 raft_consensus.cc:374] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.832396 26801 raft_consensus.cc:729] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.832439 26666 tablet_service.cc:1467] Processing CreateTablet for tablet d9f9a9d39e3a47f4b19e44cf9dcf8c34 (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20240501 19:56:00.832499 26801 consensus_queue.cc:260] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.832672 26666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9f9a9d39e3a47f4b19e44cf9dcf8c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.832619 26667 tablet_service.cc:1467] Processing CreateTablet for tablet 25fe56f01fff4c78a5b9759df9ada16e (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20240501 19:56:00.832721 26801 ts_tablet_manager.cc:1398] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.832840 26667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fe56f01fff4c78a5b9759df9ada16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.832439 26668 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6b7e7eb704466594924a582419499f (DEFAULT_TABLE table=test-workload [id=eb602928ccd74b54ba8b9327aa4b692e]), partition=RANGE (key) PARTITION VALUES < 268435455
I20240501 19:56:00.833094 26799 tablet_bootstrap.cc:492] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd: Bootstrap starting.
I20240501 19:56:00.834316 26799 tablet_bootstrap.cc:654] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.834836 26668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6b7e7eb704466594924a582419499f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:00.834986 26800 tablet_bootstrap.cc:492] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4: Bootstrap starting.
I20240501 19:56:00.835886 26800 tablet_bootstrap.cc:654] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.832887 26801 tablet_bootstrap.cc:492] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f: Bootstrap starting.
I20240501 19:56:00.837080 26799 tablet_bootstrap.cc:492] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd: No bootstrap required, opened a new log
I20240501 19:56:00.837080 26800 tablet_bootstrap.cc:492] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4: No bootstrap required, opened a new log
I20240501 19:56:00.837182 26799 ts_tablet_manager.cc:1367] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:00.837249 26800 ts_tablet_manager.cc:1367] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:00.837568 26799 raft_consensus.cc:348] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:00.837651 26799 raft_consensus.cc:374] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.837666 26801 tablet_bootstrap.cc:654] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.837684 26799 raft_consensus.cc:729] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.837677 26800 raft_consensus.cc:348] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.837865 26800 raft_consensus.cc:374] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.837852 26799 consensus_queue.cc:260] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:00.837934 26800 raft_consensus.cc:729] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.838054 26800 consensus_queue.cc:260] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.838270 26799 ts_tablet_manager.cc:1398] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:00.838426 26800 ts_tablet_manager.cc:1398] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.838874 26800 tablet_bootstrap.cc:492] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4: Bootstrap starting.
I20240501 19:56:00.839030 26801 tablet_bootstrap.cc:492] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f: No bootstrap required, opened a new log
I20240501 19:56:00.839180 26801 ts_tablet_manager.cc:1367] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:00.839802 26801 raft_consensus.cc:348] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.839916 26801 raft_consensus.cc:374] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.839968 26801 raft_consensus.cc:729] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.840088 26801 consensus_queue.cc:260] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.840261 26800 tablet_bootstrap.cc:654] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.840364 26801 ts_tablet_manager.cc:1398] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.840528 26801 tablet_bootstrap.cc:492] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f: Bootstrap starting.
I20240501 19:56:00.841001 26799 tablet_bootstrap.cc:492] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd: Bootstrap starting.
I20240501 19:56:00.841403 26801 tablet_bootstrap.cc:654] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.841420 26800 tablet_bootstrap.cc:492] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4: No bootstrap required, opened a new log
I20240501 19:56:00.841512 26800 ts_tablet_manager.cc:1367] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:00.841936 26800 raft_consensus.cc:348] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:00.842041 26800 raft_consensus.cc:374] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.842080 26800 raft_consensus.cc:729] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.842087 26799 tablet_bootstrap.cc:654] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.842269 26800 consensus_queue.cc:260] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:00.842682 26800 ts_tablet_manager.cc:1398] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.843037 26800 tablet_bootstrap.cc:492] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4: Bootstrap starting.
I20240501 19:56:00.843142 26799 tablet_bootstrap.cc:492] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd: No bootstrap required, opened a new log
I20240501 19:56:00.843230 26799 ts_tablet_manager.cc:1367] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:00.843555 26799 raft_consensus.cc:348] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.843636 26799 raft_consensus.cc:374] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.843668 26799 raft_consensus.cc:729] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.843780 26799 consensus_queue.cc:260] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.843907 26800 tablet_bootstrap.cc:654] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.844036 26801 tablet_bootstrap.cc:492] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f: No bootstrap required, opened a new log
I20240501 19:56:00.844132 26801 ts_tablet_manager.cc:1367] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:00.844040 26799 ts_tablet_manager.cc:1398] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:00.844535 26801 raft_consensus.cc:348] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.844648 26801 raft_consensus.cc:374] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.844705 26801 raft_consensus.cc:729] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.844856 26801 consensus_queue.cc:260] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.844658 26799 tablet_bootstrap.cc:492] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd: Bootstrap starting.
I20240501 19:56:00.845265 26801 ts_tablet_manager.cc:1398] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.845438 26801 tablet_bootstrap.cc:492] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f: Bootstrap starting.
I20240501 19:56:00.845819 26799 tablet_bootstrap.cc:654] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.846038 26800 tablet_bootstrap.cc:492] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4: No bootstrap required, opened a new log
I20240501 19:56:00.846155 26800 ts_tablet_manager.cc:1367] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:00.846280 26801 tablet_bootstrap.cc:654] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.846552 26800 raft_consensus.cc:348] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.846644 26800 raft_consensus.cc:374] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.846683 26800 raft_consensus.cc:729] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.846810 26800 consensus_queue.cc:260] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.847077 26800 ts_tablet_manager.cc:1398] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.847249 26800 tablet_bootstrap.cc:492] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4: Bootstrap starting.
I20240501 19:56:00.847296 26799 tablet_bootstrap.cc:492] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd: No bootstrap required, opened a new log
I20240501 19:56:00.847363 26799 ts_tablet_manager.cc:1367] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:00.847734 26799 raft_consensus.cc:348] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.847833 26799 raft_consensus.cc:374] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.847870 26799 raft_consensus.cc:729] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.848007 26799 consensus_queue.cc:260] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.848100 26800 tablet_bootstrap.cc:654] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.848341 26799 ts_tablet_manager.cc:1398] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:00.848505 26799 tablet_bootstrap.cc:492] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd: Bootstrap starting.
I20240501 19:56:00.849184 26801 tablet_bootstrap.cc:492] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f: No bootstrap required, opened a new log
I20240501 19:56:00.849273 26801 ts_tablet_manager.cc:1367] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:00.849354 26799 tablet_bootstrap.cc:654] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.849685 26801 raft_consensus.cc:348] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.849793 26801 raft_consensus.cc:374] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.849834 26801 raft_consensus.cc:729] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.849960 26801 consensus_queue.cc:260] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.850380 26801 ts_tablet_manager.cc:1398] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.850548 26801 tablet_bootstrap.cc:492] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f: Bootstrap starting.
I20240501 19:56:00.851469 26801 tablet_bootstrap.cc:654] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.851588 26800 tablet_bootstrap.cc:492] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4: No bootstrap required, opened a new log
I20240501 19:56:00.851670 26800 ts_tablet_manager.cc:1367] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:00.853068 26800 raft_consensus.cc:348] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.853241 26800 raft_consensus.cc:374] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.853284 26800 raft_consensus.cc:729] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.853358 26799 tablet_bootstrap.cc:492] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd: No bootstrap required, opened a new log
I20240501 19:56:00.853420 26800 consensus_queue.cc:260] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.853471 26799 ts_tablet_manager.cc:1367] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20240501 19:56:00.853536 26801 tablet_bootstrap.cc:492] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f: No bootstrap required, opened a new log
I20240501 19:56:00.853617 26801 ts_tablet_manager.cc:1367] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:00.853703 26800 ts_tablet_manager.cc:1398] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:00.854099 26801 raft_consensus.cc:348] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.854240 26801 raft_consensus.cc:374] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.854269 26801 raft_consensus.cc:729] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.854374 26801 consensus_queue.cc:260] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.854641 26801 ts_tablet_manager.cc:1398] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.854848 26801 tablet_bootstrap.cc:492] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f: Bootstrap starting.
I20240501 19:56:00.855167 26800 tablet_bootstrap.cc:492] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4: Bootstrap starting.
I20240501 19:56:00.856153 26801 tablet_bootstrap.cc:654] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.854070 26799 raft_consensus.cc:348] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.857347 26799 raft_consensus.cc:374] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.857439 26799 raft_consensus.cc:729] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.857652 26801 tablet_bootstrap.cc:492] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f: No bootstrap required, opened a new log
I20240501 19:56:00.857599 26799 consensus_queue.cc:260] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.857759 26801 ts_tablet_manager.cc:1367] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:00.857998 26799 ts_tablet_manager.cc:1398] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:00.858238 26801 raft_consensus.cc:348] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:00.858247 26799 tablet_bootstrap.cc:492] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd: Bootstrap starting.
I20240501 19:56:00.858392 26801 raft_consensus.cc:374] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.858430 26801 raft_consensus.cc:729] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.858563 26801 consensus_queue.cc:260] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:00.858938 26801 ts_tablet_manager.cc:1398] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.859123 26801 tablet_bootstrap.cc:492] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f: Bootstrap starting.
I20240501 19:56:00.859548 26799 tablet_bootstrap.cc:654] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.860215 26801 tablet_bootstrap.cc:654] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.856791 26800 tablet_bootstrap.cc:654] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.861685 26799 tablet_bootstrap.cc:492] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd: No bootstrap required, opened a new log
I20240501 19:56:00.861785 26799 ts_tablet_manager.cc:1367] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:00.862280 26799 raft_consensus.cc:348] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.862375 26799 raft_consensus.cc:374] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.862412 26799 raft_consensus.cc:729] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.862540 26799 consensus_queue.cc:260] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.862638 26800 tablet_bootstrap.cc:492] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4: No bootstrap required, opened a new log
I20240501 19:56:00.862754 26800 ts_tablet_manager.cc:1367] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:00.862807 26799 ts_tablet_manager.cc:1398] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.863111 26799 tablet_bootstrap.cc:492] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd: Bootstrap starting.
I20240501 19:56:00.863293 26800 raft_consensus.cc:348] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.863413 26800 raft_consensus.cc:374] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.863453 26800 raft_consensus.cc:729] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.863571 26800 consensus_queue.cc:260] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.863919 26800 ts_tablet_manager.cc:1398] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.864079 26799 tablet_bootstrap.cc:654] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.864364 26800 tablet_bootstrap.cc:492] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4: Bootstrap starting.
I20240501 19:56:00.864408 26801 tablet_bootstrap.cc:492] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f: No bootstrap required, opened a new log
I20240501 19:56:00.864496 26801 ts_tablet_manager.cc:1367] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:00.864977 26801 raft_consensus.cc:348] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.865080 26801 raft_consensus.cc:374] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.865109 26801 raft_consensus.cc:729] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.865231 26801 consensus_queue.cc:260] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.865348 26800 tablet_bootstrap.cc:654] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.865566 26801 ts_tablet_manager.cc:1398] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.866072 26799 tablet_bootstrap.cc:492] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd: No bootstrap required, opened a new log
I20240501 19:56:00.866194 26799 ts_tablet_manager.cc:1367] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:00.866446 26800 tablet_bootstrap.cc:492] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4: No bootstrap required, opened a new log
I20240501 19:56:00.866539 26800 ts_tablet_manager.cc:1367] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:00.866611 26799 raft_consensus.cc:348] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.866739 26799 raft_consensus.cc:374] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.866776 26799 raft_consensus.cc:729] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.866894 26799 consensus_queue.cc:260] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.866971 26800 raft_consensus.cc:348] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.867075 26800 raft_consensus.cc:374] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.867293 26800 raft_consensus.cc:729] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.867422 26800 consensus_queue.cc:260] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.867187 26799 ts_tablet_manager.cc:1398] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:00.867733 26799 tablet_bootstrap.cc:492] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd: Bootstrap starting.
I20240501 19:56:00.869139 26799 tablet_bootstrap.cc:654] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.869845 26800 ts_tablet_manager.cc:1398] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:00.870051 26800 tablet_bootstrap.cc:492] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4: Bootstrap starting.
I20240501 19:56:00.870880 26800 tablet_bootstrap.cc:654] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.871734 26799 tablet_bootstrap.cc:492] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd: No bootstrap required, opened a new log
I20240501 19:56:00.871824 26799 ts_tablet_manager.cc:1367] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:00.871930 26800 tablet_bootstrap.cc:492] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4: No bootstrap required, opened a new log
I20240501 19:56:00.872015 26800 ts_tablet_manager.cc:1367] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:00.872474 26799 raft_consensus.cc:348] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.872494 26800 raft_consensus.cc:348] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.872622 26799 raft_consensus.cc:374] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.872638 26800 raft_consensus.cc:374] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.872678 26799 raft_consensus.cc:729] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.872679 26800 raft_consensus.cc:729] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.872817 26799 consensus_queue.cc:260] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.872830 26800 consensus_queue.cc:260] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.873119 26799 ts_tablet_manager.cc:1398] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:00.873154 26800 ts_tablet_manager.cc:1398] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.873308 26799 tablet_bootstrap.cc:492] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd: Bootstrap starting.
I20240501 19:56:00.873311 26800 tablet_bootstrap.cc:492] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4: Bootstrap starting.
I20240501 19:56:00.874336 26800 tablet_bootstrap.cc:654] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.874370 26799 tablet_bootstrap.cc:654] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:00.875613 26799 tablet_bootstrap.cc:492] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd: No bootstrap required, opened a new log
I20240501 19:56:00.875646 26800 tablet_bootstrap.cc:492] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4: No bootstrap required, opened a new log
I20240501 19:56:00.875778 26799 ts_tablet_manager.cc:1367] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:00.875810 26800 ts_tablet_manager.cc:1367] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:00.876211 26799 raft_consensus.cc:348] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.876300 26799 raft_consensus.cc:374] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.876324 26799 raft_consensus.cc:729] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.876374 26800 raft_consensus.cc:348] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.876444 26799 consensus_queue.cc:260] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.876495 26800 raft_consensus.cc:374] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:00.876638 26800 raft_consensus.cc:729] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Initialized, Role: FOLLOWER
I20240501 19:56:00.876703 26799 ts_tablet_manager.cc:1398] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:00.876791 26800 consensus_queue.cc:260] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.877205 26800 ts_tablet_manager.cc:1398] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:00.894796 26812 raft_consensus.cc:482] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:00.894980 26812 raft_consensus.cc:504] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.895556 26812 leader_election.cc:288] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:00.897406 26812 raft_consensus.cc:482] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:00.897526 26812 raft_consensus.cc:504] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.897886 26812 leader_election.cc:288] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:00.899947 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e11c4a0fa824134a1cdd3caa6d494a7" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" is_pre_election: true
I20240501 19:56:00.900046 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e603f27d210f4d4890043cddf48679ce" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" is_pre_election: true
I20240501 19:56:00.900095 26602 raft_consensus.cc:2454] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:00.900157 26601 raft_consensus.cc:2454] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:00.900207 26803 raft_consensus.cc:482] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:00.900264 26803 raft_consensus.cc:504] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.900354 26678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e11c4a0fa824134a1cdd3caa6d494a7" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f" is_pre_election: true
I20240501 19:56:00.900431 26678 raft_consensus.cc:2454] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:00.900643 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e603f27d210f4d4890043cddf48679ce" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f" is_pre_election: true
I20240501 19:56:00.900698 26803 leader_election.cc:288] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:00.900722 26716 leader_election.cc:302] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:00.900916 26716 leader_election.cc:302] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:00.900951 26812 raft_consensus.cc:2781] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:00.901003 26812 raft_consensus.cc:482] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:00.901050 26812 raft_consensus.cc:3037] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.900741 26677 raft_consensus.cc:2454] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:00.902091 26809 raft_consensus.cc:2781] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:00.902177 26809 raft_consensus.cc:482] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:00.902232 26809 raft_consensus.cc:3037] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.905547 26753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72cd6084c9f4fbc9e881c09de7e4047" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d10550110544517b0590ca73d778ce4" is_pre_election: true
I20240501 19:56:00.905671 26753 raft_consensus.cc:2454] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 0.
I20240501 19:56:00.905966 26642 leader_election.cc:302] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, 3d10550110544517b0590ca73d778ce4; no voters:
I20240501 19:56:00.906158 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72cd6084c9f4fbc9e881c09de7e4047" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" is_pre_election: true
I20240501 19:56:00.906276 26601 raft_consensus.cc:2454] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 0.
I20240501 19:56:00.906425 26803 raft_consensus.cc:2781] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:00.906476 26803 raft_consensus.cc:482] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:00.906519 26803 raft_consensus.cc:3037] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.908972 26835 raft_consensus.cc:482] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:00.909057 26835 raft_consensus.cc:504] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.909397 26835 leader_election.cc:288] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:00.909687 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154e1e14b2d403fa4ab1fdbf2aaa972" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" is_pre_election: true
I20240501 19:56:00.909797 26601 raft_consensus.cc:2454] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:00.909976 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154e1e14b2d403fa4ab1fdbf2aaa972" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f" is_pre_election: true
I20240501 19:56:00.910076 26677 raft_consensus.cc:2454] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:00.910176 26716 leader_election.cc:302] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:00.910318 26835 raft_consensus.cc:2781] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:00.910372 26835 raft_consensus.cc:482] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:00.910406 26835 raft_consensus.cc:3037] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Advancing to term 1
W20240501 19:56:00.950798 26629 tablet.cc:2369] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:00.954334 26807 raft_consensus.cc:482] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:00.954463 26807 raft_consensus.cc:504] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.954988 26807 leader_election.cc:288] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:00.959219 26753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fe56f01fff4c78a5b9759df9ada16e" candidate_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d10550110544517b0590ca73d778ce4" is_pre_election: true
I20240501 19:56:00.959224 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fe56f01fff4c78a5b9759df9ada16e" candidate_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f" is_pre_election: true
I20240501 19:56:00.959365 26753 raft_consensus.cc:2454] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0d6032b1964c14bc138e0a28e5ccfd in term 0.
I20240501 19:56:00.959389 26677 raft_consensus.cc:2454] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df0d6032b1964c14bc138e0a28e5ccfd in term 0.
I20240501 19:56:00.959621 26566 leader_election.cc:302] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:00.959782 26807 raft_consensus.cc:2781] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:00.959841 26807 raft_consensus.cc:482] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:00.959880 26807 raft_consensus.cc:3037] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.971423 26838 raft_consensus.cc:482] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:00.971514 26838 raft_consensus.cc:504] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.971823 26838 leader_election.cc:288] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:00.972117 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f9a9d39e3a47f4b19e44cf9dcf8c34" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" is_pre_election: true
I20240501 19:56:00.972154 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f9a9d39e3a47f4b19e44cf9dcf8c34" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f" is_pre_election: true
I20240501 19:56:00.972210 26601 raft_consensus.cc:2454] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:00.972311 26677 raft_consensus.cc:2454] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:00.972483 26716 leader_election.cc:302] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:00.972628 26838 raft_consensus.cc:2781] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:00.972687 26838 raft_consensus.cc:482] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:00.972728 26838 raft_consensus.cc:3037] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.989620 26803 raft_consensus.cc:504] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.990228 26803 leader_election.cc:288] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 1 election: Requested vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:00.990788 26835 raft_consensus.cc:504] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.990934 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72cd6084c9f4fbc9e881c09de7e4047" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
I20240501 19:56:00.991058 26601 raft_consensus.cc:3037] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.991094 26835 leader_election.cc:288] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Requested vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:00.989620 26807 raft_consensus.cc:504] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.991395 26807 leader_election.cc:288] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [CANDIDATE]: Term 1 election: Requested vote from peers 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:00.991429 26753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a72cd6084c9f4fbc9e881c09de7e4047" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d10550110544517b0590ca73d778ce4"
I20240501 19:56:00.991536 26753 raft_consensus.cc:3037] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.991945 26601 raft_consensus.cc:2454] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 1.
I20240501 19:56:00.992216 26838 raft_consensus.cc:504] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.992478 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154e1e14b2d403fa4ab1fdbf2aaa972" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f"
I20240501 19:56:00.992558 26838 leader_election.cc:288] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Requested vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:00.992576 26677 raft_consensus.cc:3037] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.990878 26809 raft_consensus.cc:504] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.992622 26753 raft_consensus.cc:2454] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 1.
I20240501 19:56:00.992887 26809 leader_election.cc:288] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Requested vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:00.993077 26678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f9a9d39e3a47f4b19e44cf9dcf8c34" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f"
I20240501 19:56:00.989620 26812 raft_consensus.cc:504] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.993177 26678 raft_consensus.cc:3037] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.993419 26677 raft_consensus.cc:2454] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:00.993479 26812 leader_election.cc:288] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Requested vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:00.993801 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e603f27d210f4d4890043cddf48679ce" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f"
I20240501 19:56:00.993898 26677 raft_consensus.cc:3037] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.993913 26676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fe56f01fff4c78a5b9759df9ada16e" candidate_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f"
I20240501 19:56:00.994006 26676 raft_consensus.cc:3037] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.994424 26717 leader_election.cc:302] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, 3d10550110544517b0590ca73d778ce4; no voters:
I20240501 19:56:00.994539 26602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f9a9d39e3a47f4b19e44cf9dcf8c34" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
I20240501 19:56:00.994637 26602 raft_consensus.cc:3037] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.994684 26641 leader_election.cc:302] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:00.994767 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e11c4a0fa824134a1cdd3caa6d494a7" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
I20240501 19:56:00.994871 26599 raft_consensus.cc:3037] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.994668 26600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e603f27d210f4d4890043cddf48679ce" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
I20240501 19:56:00.995095 26675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e11c4a0fa824134a1cdd3caa6d494a7" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f"
I20240501 19:56:00.995223 26675 raft_consensus.cc:3037] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.995286 26803 raft_consensus.cc:2781] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:00.994441 26753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fe56f01fff4c78a5b9759df9ada16e" candidate_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d10550110544517b0590ca73d778ce4"
I20240501 19:56:00.995458 26753 raft_consensus.cc:3037] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.995369 26803 raft_consensus.cc:686] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 1 LEADER]: Becoming Leader. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Running, Role: LEADER
I20240501 19:56:00.995540 26601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9154e1e14b2d403fa4ab1fdbf2aaa972" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
I20240501 19:56:00.995613 26601 raft_consensus.cc:3037] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.995079 26600 raft_consensus.cc:3037] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:00.995637 26803 consensus_queue.cc:237] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.996047 26812 raft_consensus.cc:2781] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:00.996114 26812 raft_consensus.cc:686] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Running, Role: LEADER
I20240501 19:56:00.996201 26812 consensus_queue.cc:237] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:00.997504 26499 catalog_manager.cc:5484] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d10550110544517b0590ca73d778ce4 (127.25.83.3). New cstate: current_term: 1 leader_uuid: "3d10550110544517b0590ca73d778ce4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:00.997517 26497 catalog_manager.cc:5484] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f reported cstate change: term changed from 0 to 1, leader changed from <none> to 142ca6504e854a268686cc7bbc10936f (127.25.83.2). New cstate: current_term: 1 leader_uuid: "142ca6504e854a268686cc7bbc10936f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:01.031478 26803 raft_consensus.cc:482] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:01.031610 26803 raft_consensus.cc:504] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:01.032011 26803 leader_election.cc:288] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:01.032433 26752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6b7e7eb704466594924a582419499f" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d10550110544517b0590ca73d778ce4" is_pre_election: true
I20240501 19:56:01.032438 26598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6b7e7eb704466594924a582419499f" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" is_pre_election: true
I20240501 19:56:01.032549 26752 raft_consensus.cc:2454] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 0.
I20240501 19:56:01.032595 26598 raft_consensus.cc:2454] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 0.
I20240501 19:56:01.032804 26642 leader_election.cc:302] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, 3d10550110544517b0590ca73d778ce4; no voters:
I20240501 19:56:01.033149 26803 raft_consensus.cc:2781] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:01.033215 26803 raft_consensus.cc:482] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:01.033262 26803 raft_consensus.cc:3037] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:01.076681 26678 raft_consensus.cc:2454] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.076872 26677 raft_consensus.cc:2454] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.076681 26602 raft_consensus.cc:2454] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.076681 26753 raft_consensus.cc:2454] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0d6032b1964c14bc138e0a28e5ccfd in term 1.
I20240501 19:56:01.077286 26717 leader_election.cc:302] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, 3d10550110544517b0590ca73d778ce4; no voters:
I20240501 19:56:01.077494 26717 leader_election.cc:302] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, 3d10550110544517b0590ca73d778ce4; no voters:
I20240501 19:56:01.077682 26601 raft_consensus.cc:2454] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.077752 26812 raft_consensus.cc:2781] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:01.077834 26812 raft_consensus.cc:686] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Running, Role: LEADER
I20240501 19:56:01.077883 26566 leader_election.cc:302] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:01.078063 26807 raft_consensus.cc:2781] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:01.078084 26600 raft_consensus.cc:2454] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.078014 26812 consensus_queue.cc:237] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.078330 26807 raft_consensus.cc:686] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 1 LEADER]: Becoming Leader. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Running, Role: LEADER
I20240501 19:56:01.078491 26807 consensus_queue.cc:237] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.078706 26838 raft_consensus.cc:2781] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:01.079038 26599 raft_consensus.cc:2454] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.079077 26838 raft_consensus.cc:686] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Running, Role: LEADER
I20240501 19:56:01.079315 26838 consensus_queue.cc:237] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.080314 26676 raft_consensus.cc:2454] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0d6032b1964c14bc138e0a28e5ccfd in term 1.
I20240501 19:56:01.080962 26716 leader_election.cc:302] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:01.081264 26838 raft_consensus.cc:2781] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:01.081390 26838 raft_consensus.cc:686] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Running, Role: LEADER
I20240501 19:56:01.081493 26838 consensus_queue.cc:237] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.081728 26497 catalog_manager.cc:5484] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd reported cstate change: term changed from 0 to 1, leader changed from <none> to df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1). New cstate: current_term: 1 leader_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:01.082335 26499 catalog_manager.cc:5484] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d10550110544517b0590ca73d778ce4 (127.25.83.3). New cstate: current_term: 1 leader_uuid: "3d10550110544517b0590ca73d778ce4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:01.082587 26499 catalog_manager.cc:5484] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d10550110544517b0590ca73d778ce4 (127.25.83.3). New cstate: current_term: 1 leader_uuid: "3d10550110544517b0590ca73d778ce4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:01.084151 26838 raft_consensus.cc:482] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:01.084241 26838 raft_consensus.cc:504] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.084542 26838 leader_election.cc:288] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:01.084803 26676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6afb91c7039d44fba7e4b3355421a01d" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f" is_pre_election: true
I20240501 19:56:01.084789 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6afb91c7039d44fba7e4b3355421a01d" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" is_pre_election: true
I20240501 19:56:01.084924 26676 raft_consensus.cc:2454] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:01.084941 26599 raft_consensus.cc:2454] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 0.
I20240501 19:56:01.085142 26716 leader_election.cc:302] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:01.085287 26838 raft_consensus.cc:2781] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:01.085328 26838 raft_consensus.cc:482] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:01.085376 26838 raft_consensus.cc:3037] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:01.085748 26499 catalog_manager.cc:5484] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d10550110544517b0590ca73d778ce4 (127.25.83.3). New cstate: current_term: 1 leader_uuid: "3d10550110544517b0590ca73d778ce4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:01.175334 26675 raft_consensus.cc:2454] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.175343 26838 raft_consensus.cc:504] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.175478 26803 raft_consensus.cc:504] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:01.175889 26838 leader_election.cc:288] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Requested vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931)
I20240501 19:56:01.175948 26803 leader_election.cc:288] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 1 election: Requested vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:01.176383 26599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6afb91c7039d44fba7e4b3355421a01d" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
I20240501 19:56:01.176434 26600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6b7e7eb704466594924a582419499f" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
I20240501 19:56:01.176503 26600 raft_consensus.cc:3037] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:01.176623 26675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6afb91c7039d44fba7e4b3355421a01d" candidate_uuid: "3d10550110544517b0590ca73d778ce4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "142ca6504e854a268686cc7bbc10936f"
I20240501 19:56:01.176503 26599 raft_consensus.cc:3037] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:01.176734 26675 raft_consensus.cc:3037] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:01.177520 26600 raft_consensus.cc:2454] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 1.
I20240501 19:56:01.177587 26599 raft_consensus.cc:2454] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.176383 26753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6b7e7eb704466594924a582419499f" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d10550110544517b0590ca73d778ce4"
I20240501 19:56:01.177753 26753 raft_consensus.cc:3037] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:01.177865 26675 raft_consensus.cc:2454] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d10550110544517b0590ca73d778ce4 in term 1.
I20240501 19:56:01.177995 26641 leader_election.cc:302] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:01.178180 26717 leader_election.cc:302] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, 3d10550110544517b0590ca73d778ce4; no voters:
I20240501 19:56:01.178231 26803 raft_consensus.cc:2781] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:01.178301 26803 raft_consensus.cc:686] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 1 LEADER]: Becoming Leader. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Running, Role: LEADER
I20240501 19:56:01.178431 26803 consensus_queue.cc:237] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } }
I20240501 19:56:01.178627 26753 raft_consensus.cc:2454] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 1.
I20240501 19:56:01.178987 26838 raft_consensus.cc:2781] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:01.179132 26838 raft_consensus.cc:686] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Running, Role: LEADER
I20240501 19:56:01.179246 26838 consensus_queue.cc:237] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.180150 26499 catalog_manager.cc:5484] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f reported cstate change: term changed from 0 to 1, leader changed from <none> to 142ca6504e854a268686cc7bbc10936f (127.25.83.2). New cstate: current_term: 1 leader_uuid: "142ca6504e854a268686cc7bbc10936f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:01.180652 26497 catalog_manager.cc:5484] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d10550110544517b0590ca73d778ce4 (127.25.83.3). New cstate: current_term: 1 leader_uuid: "3d10550110544517b0590ca73d778ce4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:01.268841 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:01.272141 26849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:01.273223 26848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:01.273959 26851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:01.274569 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:01.274706 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:01.274739 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:01.274755 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593361274754 us; error 0 us; skew 500 ppm
I20240501 19:56:01.275733 25932 webserver.cc:460] Webserver started at http://127.25.83.4:43221/ using document root <none> and password file <none>
I20240501 19:56:01.275863 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:01.275904 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:01.275960 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:01.276347 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-3-root/instance:
uuid: "f4defd08d9374204ad490b7a25dc3305"
format_stamp: "Formatted at 2024-05-01 19:56:01 on dist-test-slave-mfp3"
I20240501 19:56:01.277750 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:01.278606 26856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:01.278825 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:01.278890 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-3-root
uuid: "f4defd08d9374204ad490b7a25dc3305"
format_stamp: "Formatted at 2024-05-01 19:56:01 on dist-test-slave-mfp3"
I20240501 19:56:01.278959 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1714593357002142-25932-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:01.291390 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:01.291690 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:01.291955 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:01.292379 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:01.292418 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:01.292460 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:01.292495 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:01.297379 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.4:42813
I20240501 19:56:01.297443 26919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.4:42813 every 8 connection(s)
I20240501 19:56:01.301492 26920 heartbeater.cc:347] Connected to a master server at 127.25.83.62:44465
I20240501 19:56:01.301594 26920 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:01.301774 26920 heartbeater.cc:511] Master 127.25.83.62:44465 requested a full tablet report, sending...
I20240501 19:56:01.302304 26497 ts_manager.cc:194] Registered new tserver with Master: f4defd08d9374204ad490b7a25dc3305 (127.25.83.4:42813)
I20240501 19:56:01.302783 26497 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58834
I20240501 19:56:01.377488 26803 consensus_queue.cc:1035] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.380416 26925 consensus_queue.cc:1035] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.434623 26838 consensus_queue.cc:1035] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.438325 26835 consensus_queue.cc:1035] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.548830 26841 consensus_queue.cc:1035] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.552060 26932 consensus_queue.cc:1035] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.579380 26835 consensus_queue.cc:1035] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.582155 26838 consensus_queue.cc:1035] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.615558 26553 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:01.620754 26753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e11c4a0fa824134a1cdd3caa6d494a7"
dest_uuid: "3d10550110544517b0590ca73d778ce4"
mode: GRACEFUL
new_leader_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
from {username='slave'} at 127.0.0.1:32820
I20240501 19:56:01.620944 26753 raft_consensus.cc:595] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Received request to transfer leadership to TS df0d6032b1964c14bc138e0a28e5ccfd
I20240501 19:56:01.621680 26753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9f9a9d39e3a47f4b19e44cf9dcf8c34"
dest_uuid: "3d10550110544517b0590ca73d778ce4"
mode: GRACEFUL
new_leader_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
from {username='slave'} at 127.0.0.1:32820
I20240501 19:56:01.621774 26753 raft_consensus.cc:595] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Received request to transfer leadership to TS df0d6032b1964c14bc138e0a28e5ccfd
I20240501 19:56:01.622417 26753 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e603f27d210f4d4890043cddf48679ce"
dest_uuid: "3d10550110544517b0590ca73d778ce4"
mode: GRACEFUL
new_leader_uuid: "142ca6504e854a268686cc7bbc10936f"
from {username='slave'} at 127.0.0.1:32820
I20240501 19:56:01.622536 26753 raft_consensus.cc:595] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Received request to transfer leadership to TS 142ca6504e854a268686cc7bbc10936f
I20240501 19:56:01.622803 26553 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 19:56:01.622992 26553 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:01.635999 26838 consensus_queue.cc:1035] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.636054 26835 consensus_queue.cc:1035] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.637527 26838 consensus_queue.cc:1035] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.637943 26838 raft_consensus.cc:979] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4: : Instructing follower df0d6032b1964c14bc138e0a28e5ccfd to start an election
I20240501 19:56:01.638043 26838 raft_consensus.cc:1067] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Signalling peer df0d6032b1964c14bc138e0a28e5ccfd to start an election
I20240501 19:56:01.638988 26601 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9f9a9d39e3a47f4b19e44cf9dcf8c34"
dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
from {username='slave'} at 127.0.0.1:54416
I20240501 19:56:01.639124 26601 raft_consensus.cc:482] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:01.639187 26601 raft_consensus.cc:3037] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:01.639441 26809 consensus_queue.cc:1035] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.640527 26809 consensus_queue.cc:1035] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.642055 26809 consensus_queue.cc:1035] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.643615 26941 raft_consensus.cc:979] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4: : Instructing follower 142ca6504e854a268686cc7bbc10936f to start an election
I20240501 19:56:01.643705 26809 raft_consensus.cc:1067] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Signalling peer 142ca6504e854a268686cc7bbc10936f to start an election
I20240501 19:56:01.644130 26677 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e603f27d210f4d4890043cddf48679ce"
dest_uuid: "142ca6504e854a268686cc7bbc10936f"
from {username='slave'} at 127.0.0.1:36226
I20240501 19:56:01.644241 26677 raft_consensus.cc:482] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:01.644305 26677 raft_consensus.cc:3037] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:01.653911 26677 raft_consensus.cc:504] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.653913 26601 raft_consensus.cc:504] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.654305 26600 raft_consensus.cc:1226] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Rejecting Update request from peer 3d10550110544517b0590ca73d778ce4 for earlier term 1. Current term is 2. Ops: []
I20240501 19:56:01.654449 26677 leader_election.cc:288] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 2 election: Requested vote from peers df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1:39375), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:01.655012 26600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e603f27d210f4d4890043cddf48679ce" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
I20240501 19:56:01.655056 26753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e603f27d210f4d4890043cddf48679ce" candidate_uuid: "142ca6504e854a268686cc7bbc10936f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d10550110544517b0590ca73d778ce4"
I20240501 19:56:01.655124 26753 raft_consensus.cc:3032] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:01.655177 26753 raft_consensus.cc:729] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Running, Role: LEADER
I20240501 19:56:01.654450 26601 leader_election.cc:288] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [CANDIDATE]: Term 2 election: Requested vote from peers 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:01.655340 26752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f9a9d39e3a47f4b19e44cf9dcf8c34" candidate_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d10550110544517b0590ca73d778ce4"
I20240501 19:56:01.655344 26753 consensus_queue.cc:260] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.655434 26752 raft_consensus.cc:3032] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:01.655468 26752 raft_consensus.cc:729] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Running, Role: LEADER
I20240501 19:56:01.655543 26753 raft_consensus.cc:3037] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:01.655597 26752 consensus_queue.cc:260] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.655718 26941 consensus_queue.cc:1046] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.655124 26600 raft_consensus.cc:3037] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:01.655997 26752 raft_consensus.cc:3037] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:01.656543 26753 raft_consensus.cc:2454] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 2.
I20240501 19:56:01.656862 26642 leader_election.cc:302] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 142ca6504e854a268686cc7bbc10936f, 3d10550110544517b0590ca73d778ce4; no voters:
I20240501 19:56:01.656910 26600 raft_consensus.cc:2454] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 142ca6504e854a268686cc7bbc10936f in term 2.
I20240501 19:56:01.657022 26815 raft_consensus.cc:2781] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:01.657101 26815 raft_consensus.cc:686] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 2 LEADER]: Becoming Leader. State: Replica: 142ca6504e854a268686cc7bbc10936f, State: Running, Role: LEADER
I20240501 19:56:01.657222 26815 consensus_queue.cc:237] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.655607 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9f9a9d39e3a47f4b19e44cf9dcf8c34" candidate_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "142ca6504e854a268686cc7bbc10936f"
I20240501 19:56:01.657666 26677 raft_consensus.cc:3037] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:01.657292 26752 raft_consensus.cc:2454] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0d6032b1964c14bc138e0a28e5ccfd in term 2.
I20240501 19:56:01.658308 26566 leader_election.cc:302] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:01.658471 26807 raft_consensus.cc:2781] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:01.658592 26807 raft_consensus.cc:686] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 LEADER]: Becoming Leader. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Running, Role: LEADER
I20240501 19:56:01.658727 26807 consensus_queue.cc:237] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:01.659060 26499 catalog_manager.cc:5484] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f reported cstate change: term changed from 1 to 2, leader changed from 3d10550110544517b0590ca73d778ce4 (127.25.83.3) to 142ca6504e854a268686cc7bbc10936f (127.25.83.2). New cstate: current_term: 2 leader_uuid: "142ca6504e854a268686cc7bbc10936f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:01.659984 26498 catalog_manager.cc:5484] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd reported cstate change: term changed from 1 to 2, leader changed from 3d10550110544517b0590ca73d778ce4 (127.25.83.3) to df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1). New cstate: current_term: 2 leader_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:01.690444 26815 consensus_queue.cc:1035] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.694204 26815 consensus_queue.cc:1035] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:01.751868 26677 raft_consensus.cc:2454] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0d6032b1964c14bc138e0a28e5ccfd in term 2.
I20240501 19:56:01.969383 26809 raft_consensus.cc:979] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4: : Instructing follower df0d6032b1964c14bc138e0a28e5ccfd to start an election
I20240501 19:56:01.969512 26809 raft_consensus.cc:1067] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Signalling peer df0d6032b1964c14bc138e0a28e5ccfd to start an election
I20240501 19:56:01.969913 26600 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e11c4a0fa824134a1cdd3caa6d494a7"
dest_uuid: "df0d6032b1964c14bc138e0a28e5ccfd"
from {username='slave'} at 127.0.0.1:54416
I20240501 19:56:01.970050 26600 raft_consensus.cc:482] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:01.970103 26600 raft_consensus.cc:3037] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:02.053371 26600 raft_consensus.cc:504] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:02.054297 26600 leader_election.cc:288] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [CANDIDATE]: Term 2 election: Requested vote from peers 142ca6504e854a268686cc7bbc10936f (127.25.83.2:38931), 3d10550110544517b0590ca73d778ce4 (127.25.83.3:44891)
I20240501 19:56:02.054898 26752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e11c4a0fa824134a1cdd3caa6d494a7" candidate_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3d10550110544517b0590ca73d778ce4"
I20240501 19:56:02.055037 26752 raft_consensus.cc:3032] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:02.055090 26752 raft_consensus.cc:729] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3d10550110544517b0590ca73d778ce4, State: Running, Role: LEADER
I20240501 19:56:02.055270 26752 consensus_queue.cc:260] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:02.055471 26752 raft_consensus.cc:3037] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:02.055518 26677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e11c4a0fa824134a1cdd3caa6d494a7" candidate_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "142ca6504e854a268686cc7bbc10936f"
I20240501 19:56:02.055985 26677 raft_consensus.cc:3037] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:02.056632 26752 raft_consensus.cc:2454] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0d6032b1964c14bc138e0a28e5ccfd in term 2.
I20240501 19:56:02.057042 26566 leader_election.cc:302] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d10550110544517b0590ca73d778ce4, df0d6032b1964c14bc138e0a28e5ccfd; no voters:
I20240501 19:56:02.057274 26807 raft_consensus.cc:2781] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:02.057368 26807 raft_consensus.cc:686] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 LEADER]: Becoming Leader. State: Replica: df0d6032b1964c14bc138e0a28e5ccfd, State: Running, Role: LEADER
I20240501 19:56:02.057511 26807 consensus_queue.cc:237] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } }
I20240501 19:56:02.059419 26498 catalog_manager.cc:5484] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd reported cstate change: term changed from 1 to 2, leader changed from 3d10550110544517b0590ca73d778ce4 (127.25.83.3) to df0d6032b1964c14bc138e0a28e5ccfd (127.25.83.1). New cstate: current_term: 2 leader_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:02.059837 26677 raft_consensus.cc:2454] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df0d6032b1964c14bc138e0a28e5ccfd in term 2.
I20240501 19:56:02.131145 26677 raft_consensus.cc:1261] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Refusing update from remote peer df0d6032b1964c14bc138e0a28e5ccfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:02.131528 26807 consensus_queue.cc:1035] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:02.133726 26752 raft_consensus.cc:1261] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Refusing update from remote peer df0d6032b1964c14bc138e0a28e5ccfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:02.134097 26947 consensus_queue.cc:1035] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:02.138204 26600 raft_consensus.cc:1261] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Refusing update from remote peer 142ca6504e854a268686cc7bbc10936f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:02.138610 26803 consensus_queue.cc:1035] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:02.140919 26752 raft_consensus.cc:1261] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Refusing update from remote peer 142ca6504e854a268686cc7bbc10936f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:02.141364 26803 consensus_queue.cc:1035] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:02.303586 26920 heartbeater.cc:503] Master 127.25.83.62:44465 was elected leader, sending a full tablet report...
I20240501 19:56:02.526680 26677 raft_consensus.cc:1261] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Refusing update from remote peer df0d6032b1964c14bc138e0a28e5ccfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:02.527082 26807 consensus_queue.cc:1035] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:02.529191 26752 raft_consensus.cc:1261] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Refusing update from remote peer df0d6032b1964c14bc138e0a28e5ccfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:02.529822 26932 consensus_queue.cc:1035] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:03.623337 26553 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:03.624114 26553 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:03.624181 26553 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:05.624483 26553 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:05.625378 26553 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:05.625479 26553 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:06.627655 26752 consensus_queue.cc:237] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } attrs { replace: true } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } }
I20240501 19:56:06.628994 26677 raft_consensus.cc:1261] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Refusing update from remote peer 3d10550110544517b0590ca73d778ce4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.629240 26600 raft_consensus.cc:1261] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Refusing update from remote peer 3d10550110544517b0590ca73d778ce4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.629350 26952 consensus_queue.cc:1035] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:06.629644 26809 consensus_queue.cc:1035] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:06.630959 26952 raft_consensus.cc:2932] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } attrs { replace: true } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.631331 26677 raft_consensus.cc:2932] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } attrs { replace: true } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.632293 26600 raft_consensus.cc:2932] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } attrs { replace: true } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.633450 26498 catalog_manager.cc:5484] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f reported cstate change: config changed from index -1 to 2, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New cstate: current_term: 1 leader_uuid: "3d10550110544517b0590ca73d778ce4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } attrs { replace: true } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
W20240501 19:56:06.634307 26716 consensus_peers.cc:485] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 -> Peer f4defd08d9374204ad490b7a25dc3305 (127.25.83.4:42813): Couldn't send request to peer f4defd08d9374204ad490b7a25dc3305. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6afb91c7039d44fba7e4b3355421a01d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:06.638034 26677 consensus_queue.cc:237] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } attrs { replace: true } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } }
I20240501 19:56:06.639364 26752 raft_consensus.cc:1261] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Refusing update from remote peer 142ca6504e854a268686cc7bbc10936f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:06.639633 26600 raft_consensus.cc:1261] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Refusing update from remote peer 142ca6504e854a268686cc7bbc10936f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:06.639925 26964 consensus_queue.cc:1035] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:06.640168 26803 consensus_queue.cc:1035] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [LEADER]: Connected to new peer: Peer: permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:06.642282 26963 raft_consensus.cc:2932] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } attrs { replace: true } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.642642 26752 raft_consensus.cc:2932] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } attrs { replace: true } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.642786 26600 raft_consensus.cc:2932] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } attrs { replace: true } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.644419 26497 catalog_manager.cc:5484] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f reported cstate change: config changed from index -1 to 3, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New cstate: current_term: 2 leader_uuid: "142ca6504e854a268686cc7bbc10936f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:06.645336 26641 consensus_peers.cc:485] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f -> Peer f4defd08d9374204ad490b7a25dc3305 (127.25.83.4:42813): Couldn't send request to peer f4defd08d9374204ad490b7a25dc3305. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e603f27d210f4d4890043cddf48679ce. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:06.648521 26600 consensus_queue.cc:237] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } }
I20240501 19:56:06.649801 26752 raft_consensus.cc:1261] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Refusing update from remote peer df0d6032b1964c14bc138e0a28e5ccfd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:06.650082 26677 raft_consensus.cc:1261] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Refusing update from remote peer df0d6032b1964c14bc138e0a28e5ccfd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:06.650324 26971 consensus_queue.cc:1035] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:06.650509 26807 consensus_queue.cc:1035] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:06.652045 26807 raft_consensus.cc:2932] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.652426 26677 raft_consensus.cc:2932] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.653129 26752 raft_consensus.cc:2932] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
W20240501 19:56:06.654342 26565 consensus_peers.cc:485] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd -> Peer f4defd08d9374204ad490b7a25dc3305 (127.25.83.4:42813): Couldn't send request to peer f4defd08d9374204ad490b7a25dc3305. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e11c4a0fa824134a1cdd3caa6d494a7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:06.654309 26499 catalog_manager.cc:5484] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd reported cstate change: config changed from index -1 to 3, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New cstate: current_term: 2 leader_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:06.655718 26600 consensus_queue.cc:237] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } }
I20240501 19:56:06.657388 26752 raft_consensus.cc:1261] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Refusing update from remote peer df0d6032b1964c14bc138e0a28e5ccfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.657435 26677 raft_consensus.cc:1261] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Refusing update from remote peer df0d6032b1964c14bc138e0a28e5ccfd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240501 19:56:06.657435 26565 consensus_peers.cc:485] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd -> Peer f4defd08d9374204ad490b7a25dc3305 (127.25.83.4:42813): Couldn't send request to peer f4defd08d9374204ad490b7a25dc3305. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25fe56f01fff4c78a5b9759df9ada16e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:06.657758 26970 consensus_queue.cc:1035] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:06.657995 26807 consensus_queue.cc:1035] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:06.659574 26970 raft_consensus.cc:2932] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.659924 26752 raft_consensus.cc:2932] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.661489 26677 raft_consensus.cc:2932] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.661760 26497 catalog_manager.cc:5484] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 reported cstate change: config changed from index -1 to 2, NON_VOTER f4defd08d9374204ad490b7a25dc3305 (127.25.83.4) added. New cstate: current_term: 1 leader_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df0d6032b1964c14bc138e0a28e5ccfd" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 39375 } } peers { permanent_uuid: "142ca6504e854a268686cc7bbc10936f" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 38931 } attrs { replace: true } } peers { permanent_uuid: "3d10550110544517b0590ca73d778ce4" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44891 } } peers { permanent_uuid: "f4defd08d9374204ad490b7a25dc3305" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 42813 } attrs { promote: true } } }
I20240501 19:56:06.821487 25932 tablet_server.cc:178] TabletServer@127.25.83.1:0 shutting down...
W20240501 19:56:06.822382 26483 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20240501 19:56:06.823395 26547 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:56:06.823472 26547 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:56:06.826237 26547 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:56:06.826310 26547 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 19:56:06.826412 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:06.826546 25932 tablet_replica.cc:330] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd: stopping tablet replica
I20240501 19:56:06.826676 25932 raft_consensus.cc:2229] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.826788 25932 raft_consensus.cc:2258] T e603f27d210f4d4890043cddf48679ce P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.827333 25932 tablet_replica.cc:330] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd: stopping tablet replica
I20240501 19:56:06.827452 25932 raft_consensus.cc:2229] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.827540 25932 raft_consensus.cc:2258] T 6afb91c7039d44fba7e4b3355421a01d P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.827960 25932 tablet_replica.cc:330] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd: stopping tablet replica
I20240501 19:56:06.828056 25932 raft_consensus.cc:2229] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.828182 25932 raft_consensus.cc:2258] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.828526 25932 tablet_replica.cc:330] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd: stopping tablet replica
I20240501 19:56:06.828622 25932 raft_consensus.cc:2229] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.828785 25932 raft_consensus.cc:2258] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.829129 25932 tablet_replica.cc:330] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd: stopping tablet replica
I20240501 19:56:06.829227 25932 raft_consensus.cc:2229] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.829401 25932 raft_consensus.cc:2258] T 25fe56f01fff4c78a5b9759df9ada16e P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.829804 25932 tablet_replica.cc:330] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd: stopping tablet replica
I20240501 19:56:06.829902 25932 raft_consensus.cc:2229] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.829993 25932 raft_consensus.cc:2258] T a72cd6084c9f4fbc9e881c09de7e4047 P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.830328 25932 tablet_replica.cc:330] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd: stopping tablet replica
I20240501 19:56:06.830417 25932 raft_consensus.cc:2229] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.830505 25932 raft_consensus.cc:2258] T 4e6b7e7eb704466594924a582419499f P df0d6032b1964c14bc138e0a28e5ccfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.830785 25932 tablet_replica.cc:330] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd: stopping tablet replica
I20240501 19:56:06.830869 25932 raft_consensus.cc:2229] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.831082 25932 raft_consensus.cc:2258] T 1e11c4a0fa824134a1cdd3caa6d494a7 P df0d6032b1964c14bc138e0a28e5ccfd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.845297 25932 tablet_server.cc:194] TabletServer@127.25.83.1:0 shutdown complete.
I20240501 19:56:06.848356 25932 tablet_server.cc:178] TabletServer@127.25.83.2:0 shutting down...
W20240501 19:56:06.849670 26547 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:56:06.849727 26547 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 19:56:06.856884 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:06.857240 25932 tablet_replica.cc:330] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f: stopping tablet replica
I20240501 19:56:06.857450 25932 raft_consensus.cc:2229] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.857625 25932 raft_consensus.cc:2258] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.858438 25932 tablet_replica.cc:330] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f: stopping tablet replica
I20240501 19:56:06.858577 25932 raft_consensus.cc:2229] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.858707 25932 raft_consensus.cc:2258] T 6afb91c7039d44fba7e4b3355421a01d P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.859292 25932 tablet_replica.cc:330] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f: stopping tablet replica
I20240501 19:56:06.859387 25932 raft_consensus.cc:2229] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 2 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.859601 25932 raft_consensus.cc:2258] T e603f27d210f4d4890043cddf48679ce P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.860810 25932 tablet_replica.cc:330] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f: stopping tablet replica
I20240501 19:56:06.861174 25932 raft_consensus.cc:2229] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.861410 25932 raft_consensus.cc:2258] T 4e6b7e7eb704466594924a582419499f P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.862062 25932 tablet_replica.cc:330] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f: stopping tablet replica
I20240501 19:56:06.862196 25932 raft_consensus.cc:2229] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.862313 25932 raft_consensus.cc:2258] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.862778 25932 tablet_replica.cc:330] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f: stopping tablet replica
I20240501 19:56:06.862882 25932 raft_consensus.cc:2229] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.862989 25932 raft_consensus.cc:2258] T 25fe56f01fff4c78a5b9759df9ada16e P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.863426 25932 tablet_replica.cc:330] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f: stopping tablet replica
I20240501 19:56:06.863557 25932 raft_consensus.cc:2229] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.863719 25932 raft_consensus.cc:2258] T a72cd6084c9f4fbc9e881c09de7e4047 P 142ca6504e854a268686cc7bbc10936f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.864034 25932 tablet_replica.cc:330] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f: stopping tablet replica
I20240501 19:56:06.864123 25932 raft_consensus.cc:2229] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.864219 25932 raft_consensus.cc:2258] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 142ca6504e854a268686cc7bbc10936f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.868310 25932 tablet_server.cc:194] TabletServer@127.25.83.2:0 shutdown complete.
I20240501 19:56:06.871642 25932 tablet_server.cc:178] TabletServer@127.25.83.3:0 shutting down...
W20240501 19:56:06.873046 26547 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20240501 19:56:06.873168 26547 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20240501 19:56:06.876056 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:06.876211 25932 tablet_replica.cc:330] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4: stopping tablet replica
I20240501 19:56:06.876320 25932 raft_consensus.cc:2229] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.876497 25932 raft_consensus.cc:2258] T 6afb91c7039d44fba7e4b3355421a01d P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.876960 25932 tablet_replica.cc:330] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4: stopping tablet replica
I20240501 19:56:06.877040 25932 raft_consensus.cc:2229] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.877166 25932 raft_consensus.cc:2258] T 9154e1e14b2d403fa4ab1fdbf2aaa972 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.877437 25932 tablet_replica.cc:330] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4: stopping tablet replica
I20240501 19:56:06.877507 25932 raft_consensus.cc:2229] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.877583 25932 raft_consensus.cc:2258] T e603f27d210f4d4890043cddf48679ce P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.877933 25932 tablet_replica.cc:330] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4: stopping tablet replica
I20240501 19:56:06.878000 25932 raft_consensus.cc:2229] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.878065 25932 raft_consensus.cc:2258] T 4e6b7e7eb704466594924a582419499f P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.878434 25932 tablet_replica.cc:330] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4: stopping tablet replica
I20240501 19:56:06.878502 25932 raft_consensus.cc:2229] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.878577 25932 raft_consensus.cc:2258] T 1e11c4a0fa824134a1cdd3caa6d494a7 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.878957 25932 tablet_replica.cc:330] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4: stopping tablet replica
I20240501 19:56:06.879022 25932 raft_consensus.cc:2229] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.879101 25932 raft_consensus.cc:2258] T 25fe56f01fff4c78a5b9759df9ada16e P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.879518 25932 tablet_replica.cc:330] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4: stopping tablet replica
I20240501 19:56:06.879586 25932 raft_consensus.cc:2229] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.879660 25932 raft_consensus.cc:2258] T d9f9a9d39e3a47f4b19e44cf9dcf8c34 P 3d10550110544517b0590ca73d778ce4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.879930 25932 tablet_replica.cc:330] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4: stopping tablet replica
I20240501 19:56:06.879987 25932 raft_consensus.cc:2229] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:06.880050 25932 raft_consensus.cc:2258] T a72cd6084c9f4fbc9e881c09de7e4047 P 3d10550110544517b0590ca73d778ce4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.893164 25932 tablet_server.cc:194] TabletServer@127.25.83.3:0 shutdown complete.
I20240501 19:56:06.895432 25932 tablet_server.cc:178] TabletServer@127.25.83.4:0 shutting down...
I20240501 19:56:06.898818 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:06.911435 25932 tablet_server.cc:194] TabletServer@127.25.83.4:0 shutdown complete.
I20240501 19:56:06.913100 25932 master.cc:537] Master@127.25.83.62:44465 shutting down...
I20240501 19:56:06.915937 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:06.916113 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:06.916179 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22d7a02c52524429a61b4f8838e31adc: stopping tablet replica
I20240501 19:56:06.928491 25932 master.cc:552] Master@127.25.83.62:44465 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7383 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20240501 19:56:06.938696 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:45329,127.25.83.61:33593,127.25.83.60:36667
I20240501 19:56:06.939102 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:06.941730 26981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.942714 26982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.943472 26984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.944101 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:06.944247 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:06.944280 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:06.944294 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593366944294 us; error 0 us; skew 500 ppm
I20240501 19:56:06.945156 25932 webserver.cc:460] Webserver started at http://127.25.83.62:43085/ using document root <none> and password file <none>
I20240501 19:56:06.945292 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.945331 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.945386 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:06.945713 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "ceb9e82c08394b9b8148930e35bd3533"
format_stamp: "Formatted at 2024-05-01 19:56:06 on dist-test-slave-mfp3"
I20240501 19:56:06.947132 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:06.947950 26989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.948192 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:06.948263 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "ceb9e82c08394b9b8148930e35bd3533"
format_stamp: "Formatted at 2024-05-01 19:56:06 on dist-test-slave-mfp3"
I20240501 19:56:06.948328 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.963462 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.963820 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.968147 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:45329
I20240501 19:56:06.969411 27041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:45329 every 8 connection(s)
I20240501 19:56:06.969578 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:56:06.974054 27042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.975448 27042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:56:06.976068 27044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.976758 27047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:06.977411 27045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:06.979604 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:06.979737 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:06.979784 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:06.979807 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593366979806 us; error 0 us; skew 500 ppm
I20240501 19:56:06.980465 27042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:56:06.980733 25932 webserver.cc:460] Webserver started at http://127.25.83.61:46145/ using document root <none> and password file <none>
I20240501 19:56:06.980870 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:06.980921 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:06.980993 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:06.981384 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root/instance:
uuid: "8b70afaac0df4baba858606cdf8261c7"
format_stamp: "Formatted at 2024-05-01 19:56:06 on dist-test-slave-mfp3"
W20240501 19:56:06.981652 27042 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.83.61:33593: Network error: Client connection negotiation failed: client connection to 127.25.83.61:33593: connect: Connection refused (error 111)
I20240501 19:56:06.982851 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:06.983780 27055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:06.984032 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:06.984091 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root
uuid: "8b70afaac0df4baba858606cdf8261c7"
format_stamp: "Formatted at 2024-05-01 19:56:06 on dist-test-slave-mfp3"
I20240501 19:56:06.984145 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:06.992926 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:06.993223 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:06.996738 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.61:33593
I20240501 19:56:06.996758 27107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.61:33593 every 8 connection(s)
I20240501 19:56:06.997292 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:56:06.997453 27108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.998800 27108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:56:06.999805 27111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:07.000430 27110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:07.001150 27113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:07.001741 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:07.001910 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:07.001946 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:07.001968 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593367001968 us; error 0 us; skew 500 ppm
I20240501 19:56:07.002977 25932 webserver.cc:460] Webserver started at http://127.25.83.60:33469/ using document root <none> and password file <none>
I20240501 19:56:07.003024 27108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:56:07.003127 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:07.003186 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:07.003295 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:07.003700 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root/instance:
uuid: "e7369342d1c943a18c4de59daff48ca9"
format_stamp: "Formatted at 2024-05-01 19:56:06 on dist-test-slave-mfp3"
I20240501 19:56:07.005367 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:07.006343 27120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.006574 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.006640 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root
uuid: "e7369342d1c943a18c4de59daff48ca9"
format_stamp: "Formatted at 2024-05-01 19:56:06 on dist-test-slave-mfp3"
I20240501 19:56:07.006700 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:07.007068 27108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:56:07.008291 27108 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.83.60:36667: Network error: Client connection negotiation failed: client connection to 127.25.83.60:36667: connect: Connection refused (error 111)
I20240501 19:56:07.017799 27042 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } attempt: 1
I20240501 19:56:07.020434 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:07.020798 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:07.021457 27042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
W20240501 19:56:07.022789 27042 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.25.83.60:36667: Network error: Client connection negotiation failed: client connection to 127.25.83.60:36667: connect: Connection refused (error 111)
I20240501 19:56:07.025166 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.60:36667
I20240501 19:56:07.025208 27173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.60:36667 every 8 connection(s)
I20240501 19:56:07.025580 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:56:07.025944 27174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.026934 27174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:56:07.030561 27174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:56:07.034024 27174 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
I20240501 19:56:07.038774 27174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9: Bootstrap starting.
I20240501 19:56:07.039522 27174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.040473 27174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9: No bootstrap required, opened a new log
I20240501 19:56:07.040796 27174 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.040885 27174 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.040907 27174 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7369342d1c943a18c4de59daff48ca9, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.041002 27174 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.041249 27179 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.041347 27179 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:07.041393 27174 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: configured and running, proceeding with master startup.
W20240501 19:56:07.043323 27190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:07.043380 27190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:07.073472 27108 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } attempt: 1
I20240501 19:56:07.074049 27042 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } attempt: 1
I20240501 19:56:07.079154 27108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: Bootstrap starting.
I20240501 19:56:07.079154 27042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533: Bootstrap starting.
I20240501 19:56:07.080127 27042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.080127 27108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.081344 27042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533: No bootstrap required, opened a new log
I20240501 19:56:07.081379 27108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: No bootstrap required, opened a new log
I20240501 19:56:07.081810 27108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.081810 27042 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.081956 27108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.081965 27042 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.082001 27108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b70afaac0df4baba858606cdf8261c7, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.082036 27042 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb9e82c08394b9b8148930e35bd3533, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.082173 27108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.082173 27042 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.082544 27195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.082643 27195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:07.082701 27042 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:07.082701 27108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:07.082795 27194 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.082885 27194 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:07.084833 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20240501 19:56:07.084884 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20240501 19:56:07.085309 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:07.085402 27217 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:07.085464 27217 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 19:56:07.085906 27216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:07.085959 27216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20240501 19:56:07.088184 27219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:07.088836 27221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:07.089455 27218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:07.090153 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:07.090317 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:07.090354 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:07.090369 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593367090369 us; error 0 us; skew 500 ppm
I20240501 19:56:07.091295 25932 webserver.cc:460] Webserver started at http://127.25.83.1:35285/ using document root <none> and password file <none>
I20240501 19:56:07.091414 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:07.091452 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:07.091506 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:07.091809 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root/instance:
uuid: "e05a078da6a84e0db2155dda659eae7c"
format_stamp: "Formatted at 2024-05-01 19:56:07 on dist-test-slave-mfp3"
I20240501 19:56:07.093104 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:07.093896 27226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.094158 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:07.094230 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root
uuid: "e05a078da6a84e0db2155dda659eae7c"
format_stamp: "Formatted at 2024-05-01 19:56:07 on dist-test-slave-mfp3"
I20240501 19:56:07.094300 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:07.094395 27194 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:07.094477 27194 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.094864 27194 leader_election.cc:288] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b70afaac0df4baba858606cdf8261c7 (127.25.83.61:33593), e7369342d1c943a18c4de59daff48ca9 (127.25.83.60:36667)
I20240501 19:56:07.095212 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb9e82c08394b9b8148930e35bd3533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7369342d1c943a18c4de59daff48ca9" is_pre_election: true
I20240501 19:56:07.095351 27148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb9e82c08394b9b8148930e35bd3533 in term 0.
I20240501 19:56:07.095225 27082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb9e82c08394b9b8148930e35bd3533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b70afaac0df4baba858606cdf8261c7" is_pre_election: true
I20240501 19:56:07.095598 27082 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ceb9e82c08394b9b8148930e35bd3533 in term 0.
I20240501 19:56:07.095616 26992 leader_election.cc:302] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ceb9e82c08394b9b8148930e35bd3533, e7369342d1c943a18c4de59daff48ca9; no voters:
I20240501 19:56:07.095832 27194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:07.095887 27194 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:07.095928 27194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.096627 27194 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.096890 27194 leader_election.cc:288] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [CANDIDATE]: Term 1 election: Requested vote from peers 8b70afaac0df4baba858606cdf8261c7 (127.25.83.61:33593), e7369342d1c943a18c4de59daff48ca9 (127.25.83.60:36667)
I20240501 19:56:07.097139 27082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb9e82c08394b9b8148930e35bd3533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b70afaac0df4baba858606cdf8261c7"
I20240501 19:56:07.097230 27082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.097334 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ceb9e82c08394b9b8148930e35bd3533" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7369342d1c943a18c4de59daff48ca9"
I20240501 19:56:07.097419 27148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.097844 27082 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb9e82c08394b9b8148930e35bd3533 in term 1.
I20240501 19:56:07.098136 26993 leader_election.cc:302] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b70afaac0df4baba858606cdf8261c7, ceb9e82c08394b9b8148930e35bd3533; no voters:
I20240501 19:56:07.098265 27148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ceb9e82c08394b9b8148930e35bd3533 in term 1.
I20240501 19:56:07.098304 27194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:07.098446 27194 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 1 LEADER]: Becoming Leader. State: Replica: ceb9e82c08394b9b8148930e35bd3533, State: Running, Role: LEADER
I20240501 19:56:07.098582 27194 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:07.099202 27231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb9e82c08394b9b8148930e35bd3533. Latest consensus state: current_term: 1 leader_uuid: "ceb9e82c08394b9b8148930e35bd3533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.099298 27231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:07.099536 27233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:07.100142 27233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:07.101611 27148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 1 FOLLOWER]: Refusing update from remote peer ceb9e82c08394b9b8148930e35bd3533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.101670 27082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 1 FOLLOWER]: Refusing update from remote peer ceb9e82c08394b9b8148930e35bd3533: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.101967 27194 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.102073 27231 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.103344 27195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb9e82c08394b9b8148930e35bd3533. Latest consensus state: current_term: 1 leader_uuid: "ceb9e82c08394b9b8148930e35bd3533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.103444 27195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:07.103348 27179 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceb9e82c08394b9b8148930e35bd3533. Latest consensus state: current_term: 1 leader_uuid: "ceb9e82c08394b9b8148930e35bd3533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.103498 27179 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:07.104353 27231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ceb9e82c08394b9b8148930e35bd3533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.104436 27231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:07.104504 27234 mvcc.cc:205] Tried to move back new op lower bound from 7022974431645536256 to 7022974431637180416. Current Snapshot: MvccSnapshot[applied={T|T < 7022974431645536256}]
I20240501 19:56:07.104853 27231 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ceb9e82c08394b9b8148930e35bd3533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.104890 27179 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ceb9e82c08394b9b8148930e35bd3533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.104946 27231 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:07.104950 27179 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:07.105698 27195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ceb9e82c08394b9b8148930e35bd3533" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:07.105784 27195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:07.106401 27233 catalog_manager.cc:1347] Generated new cluster ID: 88fe7d5d274d4dc9905de8ac0315d480
I20240501 19:56:07.106454 27233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:07.114953 27233 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:07.115468 27233 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:07.119581 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:07.119916 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:07.120198 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:07.120644 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:07.120684 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.120728 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:07.120754 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.126144 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.1:32783
I20240501 19:56:07.126389 27301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.1:32783 every 8 connection(s)
I20240501 19:56:07.127102 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:56:07.127677 27233 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533: Generated new TSK 0
I20240501 19:56:07.127815 27233 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:07.131511 27302 heartbeater.cc:347] Connected to a master server at 127.25.83.60:36667
I20240501 19:56:07.131587 27302 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.131759 27302 heartbeater.cc:511] Master 127.25.83.60:36667 requested a full tablet report, sending...
W20240501 19:56:07.132302 27310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:07.132381 27308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:07.132553 27138 ts_manager.cc:194] Registered new tserver with Master: e05a078da6a84e0db2155dda659eae7c (127.25.83.1:32783)
I20240501 19:56:07.132836 25932 server_base.cc:1034] running on GCE node
W20240501 19:56:07.133414 27307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:07.133630 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 19:56:07.133673 27304 heartbeater.cc:347] Connected to a master server at 127.25.83.61:33593
I20240501 19:56:07.133721 27303 heartbeater.cc:347] Connected to a master server at 127.25.83.62:45329
W20240501 19:56:07.133687 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:07.133764 27304 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.133786 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593367133786 us; error 0 us; skew 500 ppm
I20240501 19:56:07.133802 27303 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.133913 27304 heartbeater.cc:511] Master 127.25.83.61:33593 requested a full tablet report, sending...
I20240501 19:56:07.133994 27303 heartbeater.cc:511] Master 127.25.83.62:45329 requested a full tablet report, sending...
I20240501 19:56:07.134610 27006 ts_manager.cc:194] Registered new tserver with Master: e05a078da6a84e0db2155dda659eae7c (127.25.83.1:32783)
I20240501 19:56:07.134613 27072 ts_manager.cc:194] Registered new tserver with Master: e05a078da6a84e0db2155dda659eae7c (127.25.83.1:32783)
I20240501 19:56:07.134899 25932 webserver.cc:460] Webserver started at http://127.25.83.2:35365/ using document root <none> and password file <none>
I20240501 19:56:07.135051 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:07.135093 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:07.135140 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:07.135468 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root/instance:
uuid: "151e4fef4c614e559ac8dbbd249fc70a"
format_stamp: "Formatted at 2024-05-01 19:56:07 on dist-test-slave-mfp3"
I20240501 19:56:07.135474 27006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46124
I20240501 19:56:07.136970 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:07.137825 27317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.138130 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.138202 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root
uuid: "151e4fef4c614e559ac8dbbd249fc70a"
format_stamp: "Formatted at 2024-05-01 19:56:07 on dist-test-slave-mfp3"
I20240501 19:56:07.138270 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:07.153909 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:07.154362 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:07.154641 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:07.155104 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:07.155143 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.155172 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:07.155186 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.160204 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.2:46529
I20240501 19:56:07.160252 27380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.2:46529 every 8 connection(s)
I20240501 19:56:07.161170 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:07.164682 27388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:07.165314 27391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:07.166008 27381 heartbeater.cc:347] Connected to a master server at 127.25.83.60:36667
I20240501 19:56:07.166131 27381 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.166280 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:07.166410 27381 heartbeater.cc:511] Master 127.25.83.60:36667 requested a full tablet report, sending...
I20240501 19:56:07.166126 27383 heartbeater.cc:347] Connected to a master server at 127.25.83.61:33593
I20240501 19:56:07.166710 27382 heartbeater.cc:347] Connected to a master server at 127.25.83.62:45329
I20240501 19:56:07.166740 27383 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.166802 27382 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.166929 27383 heartbeater.cc:511] Master 127.25.83.61:33593 requested a full tablet report, sending...
I20240501 19:56:07.167044 27382 heartbeater.cc:511] Master 127.25.83.62:45329 requested a full tablet report, sending...
I20240501 19:56:07.167133 27138 ts_manager.cc:194] Registered new tserver with Master: 151e4fef4c614e559ac8dbbd249fc70a (127.25.83.2:46529)
I20240501 19:56:07.167533 27072 ts_manager.cc:194] Registered new tserver with Master: 151e4fef4c614e559ac8dbbd249fc70a (127.25.83.2:46529)
W20240501 19:56:07.167515 27389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:07.167726 27006 ts_manager.cc:194] Registered new tserver with Master: 151e4fef4c614e559ac8dbbd249fc70a (127.25.83.2:46529)
I20240501 19:56:07.167835 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:07.167878 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:07.167905 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593367167905 us; error 0 us; skew 500 ppm
I20240501 19:56:07.168396 27006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46138
I20240501 19:56:07.168962 25932 webserver.cc:460] Webserver started at http://127.25.83.3:39497/ using document root <none> and password file <none>
I20240501 19:56:07.169127 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:07.169194 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:07.169255 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:07.169592 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root/instance:
uuid: "b4e5d69334f14353a6a915fb7f1ebb75"
format_stamp: "Formatted at 2024-05-01 19:56:07 on dist-test-slave-mfp3"
I20240501 19:56:07.171101 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:07.172047 27396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.172318 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.172386 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root
uuid: "b4e5d69334f14353a6a915fb7f1ebb75"
format_stamp: "Formatted at 2024-05-01 19:56:07 on dist-test-slave-mfp3"
I20240501 19:56:07.172447 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1714593357002142-25932-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:07.183936 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:07.184325 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:07.184578 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:07.185102 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:07.185153 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.185191 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:07.185227 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:07.189496 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.3:35055
I20240501 19:56:07.189572 27459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.3:35055 every 8 connection(s)
I20240501 19:56:07.194487 27462 heartbeater.cc:347] Connected to a master server at 127.25.83.61:33593
I20240501 19:56:07.194579 27462 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.194739 27461 heartbeater.cc:347] Connected to a master server at 127.25.83.62:45329
I20240501 19:56:07.194810 27462 heartbeater.cc:511] Master 127.25.83.61:33593 requested a full tablet report, sending...
I20240501 19:56:07.194835 27461 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.194947 27460 heartbeater.cc:347] Connected to a master server at 127.25.83.60:36667
I20240501 19:56:07.195017 27460 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:07.195199 27460 heartbeater.cc:511] Master 127.25.83.60:36667 requested a full tablet report, sending...
I20240501 19:56:07.195279 27461 heartbeater.cc:511] Master 127.25.83.62:45329 requested a full tablet report, sending...
I20240501 19:56:07.195441 27072 ts_manager.cc:194] Registered new tserver with Master: b4e5d69334f14353a6a915fb7f1ebb75 (127.25.83.3:35055)
I20240501 19:56:07.195695 27138 ts_manager.cc:194] Registered new tserver with Master: b4e5d69334f14353a6a915fb7f1ebb75 (127.25.83.3:35055)
I20240501 19:56:07.196096 27006 ts_manager.cc:194] Registered new tserver with Master: b4e5d69334f14353a6a915fb7f1ebb75 (127.25.83.3:35055)
I20240501 19:56:07.196540 27006 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46154
I20240501 19:56:07.196715 25932 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006764341s
I20240501 19:56:08.044193 27190 catalog_manager.cc:1259] Loaded cluster ID: 88fe7d5d274d4dc9905de8ac0315d480
I20240501 19:56:08.044279 27190 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9: loading cluster ID for follower catalog manager: success
I20240501 19:56:08.045437 27190 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9: acquiring CA information for follower catalog manager: success
I20240501 19:56:08.045950 27190 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:56:08.086635 27216 catalog_manager.cc:1259] Loaded cluster ID: 88fe7d5d274d4dc9905de8ac0315d480
I20240501 19:56:08.086724 27216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: loading cluster ID for follower catalog manager: success
I20240501 19:56:08.087847 27216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: acquiring CA information for follower catalog manager: success
I20240501 19:56:08.088375 27216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240501 19:56:08.136412 27303 heartbeater.cc:503] Master 127.25.83.62:45329 was elected leader, sending a full tablet report...
I20240501 19:56:08.169242 27382 heartbeater.cc:503] Master 127.25.83.62:45329 was elected leader, sending a full tablet report...
I20240501 19:56:08.197278 27461 heartbeater.cc:503] Master 127.25.83.62:45329 was elected leader, sending a full tablet report...
I20240501 19:56:09.084944 27215 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:09.248854 25932 test_util.cc:268] Using random seed: -524987310
I20240501 19:56:09.260408 27006 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40806:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:56:09.274012 27345 tablet_service.cc:1467] Processing CreateTablet for tablet 2550edc71e5545eaa3f154ed69764d6f (DEFAULT_TABLE table=test-workload [id=bba51bb011f644dea9102cd72be7e2db]), partition=RANGE (key) PARTITION VALUES < 715827882
I20240501 19:56:09.274410 27345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2550edc71e5545eaa3f154ed69764d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.274698 27266 tablet_service.cc:1467] Processing CreateTablet for tablet d15e5694a6eb4c20906892f761ebffcb (DEFAULT_TABLE table=test-workload [id=bba51bb011f644dea9102cd72be7e2db]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20240501 19:56:09.274938 27266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d15e5694a6eb4c20906892f761ebffcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.275406 27424 tablet_service.cc:1467] Processing CreateTablet for tablet ac11c808c56c4839b8549fc136c638b1 (DEFAULT_TABLE table=test-workload [id=bba51bb011f644dea9102cd72be7e2db]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20240501 19:56:09.275648 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac11c808c56c4839b8549fc136c638b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.276859 27497 tablet_bootstrap.cc:492] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c: Bootstrap starting.
I20240501 19:56:09.277818 27497 tablet_bootstrap.cc:654] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.277976 27498 tablet_bootstrap.cc:492] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75: Bootstrap starting.
I20240501 19:56:09.278245 27496 tablet_bootstrap.cc:492] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a: Bootstrap starting.
I20240501 19:56:09.278831 27498 tablet_bootstrap.cc:654] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.278916 27497 tablet_bootstrap.cc:492] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c: No bootstrap required, opened a new log
I20240501 19:56:09.279011 27497 ts_tablet_manager.cc:1367] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:09.279040 27496 tablet_bootstrap.cc:654] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.279407 27497 raft_consensus.cc:348] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a078da6a84e0db2155dda659eae7c" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 32783 } }
I20240501 19:56:09.279584 27497 raft_consensus.cc:374] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.279650 27497 raft_consensus.cc:729] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05a078da6a84e0db2155dda659eae7c, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.279983 27496 tablet_bootstrap.cc:492] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a: No bootstrap required, opened a new log
I20240501 19:56:09.279940 27497 consensus_queue.cc:260] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a078da6a84e0db2155dda659eae7c" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 32783 } }
I20240501 19:56:09.280068 27496 ts_tablet_manager.cc:1367] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:09.280088 27497 raft_consensus.cc:388] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:09.280197 27497 raft_consensus.cc:482] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:09.280335 27497 raft_consensus.cc:3037] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.280452 27496 raft_consensus.cc:348] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151e4fef4c614e559ac8dbbd249fc70a" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46529 } }
I20240501 19:56:09.280547 27496 raft_consensus.cc:374] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.280579 27496 raft_consensus.cc:729] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 151e4fef4c614e559ac8dbbd249fc70a, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.280705 27496 consensus_queue.cc:260] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151e4fef4c614e559ac8dbbd249fc70a" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46529 } }
I20240501 19:56:09.280792 27496 raft_consensus.cc:388] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:09.280835 27496 raft_consensus.cc:482] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:09.280874 27498 tablet_bootstrap.cc:492] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75: No bootstrap required, opened a new log
I20240501 19:56:09.280887 27496 raft_consensus.cc:3037] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.280972 27498 ts_tablet_manager.cc:1367] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:09.281246 27497 raft_consensus.cc:504] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a078da6a84e0db2155dda659eae7c" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 32783 } }
I20240501 19:56:09.281383 27497 leader_election.cc:302] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05a078da6a84e0db2155dda659eae7c; no voters:
I20240501 19:56:09.281417 27498 raft_consensus.cc:348] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5d69334f14353a6a915fb7f1ebb75" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35055 } }
I20240501 19:56:09.281489 27498 raft_consensus.cc:374] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.281528 27498 raft_consensus.cc:729] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e5d69334f14353a6a915fb7f1ebb75, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.281575 27497 leader_election.cc:288] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:09.281608 27496 raft_consensus.cc:504] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151e4fef4c614e559ac8dbbd249fc70a" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46529 } }
I20240501 19:56:09.281661 27502 raft_consensus.cc:2781] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:09.281634 27498 consensus_queue.cc:260] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5d69334f14353a6a915fb7f1ebb75" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35055 } }
I20240501 19:56:09.281723 27498 raft_consensus.cc:388] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:09.281795 27498 raft_consensus.cc:482] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:09.281821 27502 raft_consensus.cc:686] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 1 LEADER]: Becoming Leader. State: Replica: e05a078da6a84e0db2155dda659eae7c, State: Running, Role: LEADER
I20240501 19:56:09.281837 27498 raft_consensus.cc:3037] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.281869 27497 ts_tablet_manager.cc:1398] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:09.281939 27502 consensus_queue.cc:237] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a078da6a84e0db2155dda659eae7c" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 32783 } }
I20240501 19:56:09.282647 27498 raft_consensus.cc:504] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5d69334f14353a6a915fb7f1ebb75" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35055 } }
I20240501 19:56:09.282756 27498 leader_election.cc:302] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e5d69334f14353a6a915fb7f1ebb75; no voters:
I20240501 19:56:09.282914 27498 leader_election.cc:288] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:09.282994 27506 raft_consensus.cc:2781] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:09.283208 27498 ts_tablet_manager.cc:1398] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:09.283344 27506 raft_consensus.cc:686] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 1 LEADER]: Becoming Leader. State: Replica: b4e5d69334f14353a6a915fb7f1ebb75, State: Running, Role: LEADER
I20240501 19:56:09.283447 27506 consensus_queue.cc:237] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5d69334f14353a6a915fb7f1ebb75" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35055 } }
I20240501 19:56:09.284395 27006 catalog_manager.cc:5484] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c reported cstate change: term changed from 0 to 1, leader changed from <none> to e05a078da6a84e0db2155dda659eae7c (127.25.83.1). New cstate: current_term: 1 leader_uuid: "e05a078da6a84e0db2155dda659eae7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a078da6a84e0db2155dda659eae7c" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 32783 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.285584 27005 catalog_manager.cc:5484] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4e5d69334f14353a6a915fb7f1ebb75 (127.25.83.3). New cstate: current_term: 1 leader_uuid: "b4e5d69334f14353a6a915fb7f1ebb75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e5d69334f14353a6a915fb7f1ebb75" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35055 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.281723 27496 leader_election.cc:302] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 151e4fef4c614e559ac8dbbd249fc70a; no voters:
I20240501 19:56:09.287228 27496 leader_election.cc:288] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:09.287312 27510 raft_consensus.cc:2781] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:09.287487 27510 raft_consensus.cc:686] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 1 LEADER]: Becoming Leader. State: Replica: 151e4fef4c614e559ac8dbbd249fc70a, State: Running, Role: LEADER
I20240501 19:56:09.287575 27496 ts_tablet_manager.cc:1398] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:09.287627 27510 consensus_queue.cc:237] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151e4fef4c614e559ac8dbbd249fc70a" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46529 } }
I20240501 19:56:09.288758 27006 catalog_manager.cc:5484] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a reported cstate change: term changed from 0 to 1, leader changed from <none> to 151e4fef4c614e559ac8dbbd249fc70a (127.25.83.2). New cstate: current_term: 1 leader_uuid: "151e4fef4c614e559ac8dbbd249fc70a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "151e4fef4c614e559ac8dbbd249fc70a" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46529 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.301024 25932 master.cc:537] Master@127.25.83.62:45329 shutting down...
I20240501 19:56:09.304075 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:09.304261 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:09.304320 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P ceb9e82c08394b9b8148930e35bd3533: stopping tablet replica
I20240501 19:56:09.317750 25932 master.cc:552] Master@127.25.83.62:45329 shutdown complete.
W20240501 19:56:10.289165 27303 heartbeater.cc:644] Failed to heartbeat to 127.25.83.62:45329 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111)
I20240501 19:56:10.794859 27520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ceb9e82c08394b9b8148930e35bd3533)
I20240501 19:56:10.794878 27521 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ceb9e82c08394b9b8148930e35bd3533)
I20240501 19:56:10.795039 27520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:10.795120 27521 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:10.795522 27520 leader_election.cc:288] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329), e7369342d1c943a18c4de59daff48ca9 (127.25.83.60:36667)
I20240501 19:56:10.795874 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b70afaac0df4baba858606cdf8261c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e7369342d1c943a18c4de59daff48ca9" is_pre_election: true
I20240501 19:56:10.795985 27148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b70afaac0df4baba858606cdf8261c7 in term 1.
I20240501 19:56:10.796159 27082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7369342d1c943a18c4de59daff48ca9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "8b70afaac0df4baba858606cdf8261c7" is_pre_election: true
I20240501 19:56:10.796232 27521 leader_election.cc:288] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329), 8b70afaac0df4baba858606cdf8261c7 (127.25.83.61:33593)
I20240501 19:56:10.796267 27082 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7369342d1c943a18c4de59daff48ca9 in term 1.
I20240501 19:56:10.796283 27058 leader_election.cc:302] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b70afaac0df4baba858606cdf8261c7, e7369342d1c943a18c4de59daff48ca9; no voters:
I20240501 19:56:10.796483 27520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:10.796531 27520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ceb9e82c08394b9b8148930e35bd3533)
I20240501 19:56:10.796563 27520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:10.796696 27124 leader_election.cc:302] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b70afaac0df4baba858606cdf8261c7, e7369342d1c943a18c4de59daff48ca9; no voters:
I20240501 19:56:10.796844 27521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:10.796903 27521 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ceb9e82c08394b9b8148930e35bd3533)
I20240501 19:56:10.796941 27521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:10.796857 27124 leader_election.cc:334] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329): Network error: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111)
W20240501 19:56:10.797089 27059 leader_election.cc:334] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329): Network error: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111)
I20240501 19:56:10.797667 27520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:10.797864 27521 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:10.797942 27520 leader_election.cc:288] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 2 election: Requested vote from peers ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329), e7369342d1c943a18c4de59daff48ca9 (127.25.83.60:36667)
I20240501 19:56:10.798238 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b70afaac0df4baba858606cdf8261c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e7369342d1c943a18c4de59daff48ca9"
I20240501 19:56:10.798278 27521 leader_election.cc:288] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [CANDIDATE]: Term 2 election: Requested vote from peers ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329), 8b70afaac0df4baba858606cdf8261c7 (127.25.83.61:33593)
I20240501 19:56:10.798393 27148 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b70afaac0df4baba858606cdf8261c7 in current term 2: Already voted for candidate e7369342d1c943a18c4de59daff48ca9 in this term.
I20240501 19:56:10.798838 27082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7369342d1c943a18c4de59daff48ca9" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "8b70afaac0df4baba858606cdf8261c7"
I20240501 19:56:10.798967 27082 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e7369342d1c943a18c4de59daff48ca9 in current term 2: Already voted for candidate 8b70afaac0df4baba858606cdf8261c7 in this term.
W20240501 19:56:10.799172 27059 leader_election.cc:334] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329): Network error: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111)
W20240501 19:56:10.799188 27124 leader_election.cc:334] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329): Network error: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111)
I20240501 19:56:10.799248 27059 leader_election.cc:302] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b70afaac0df4baba858606cdf8261c7; no voters: ceb9e82c08394b9b8148930e35bd3533, e7369342d1c943a18c4de59daff48ca9
I20240501 19:56:10.799250 27124 leader_election.cc:302] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7369342d1c943a18c4de59daff48ca9; no voters: 8b70afaac0df4baba858606cdf8261c7, ceb9e82c08394b9b8148930e35bd3533
I20240501 19:56:10.799386 27520 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 19:56:10.799396 27521 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 19:56:12.670289 27533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:12.670433 27533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:12.670827 27533 leader_election.cc:288] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329), e7369342d1c943a18c4de59daff48ca9 (127.25.83.60:36667)
I20240501 19:56:12.671228 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b70afaac0df4baba858606cdf8261c7" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e7369342d1c943a18c4de59daff48ca9" is_pre_election: true
I20240501 19:56:12.671344 27148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b70afaac0df4baba858606cdf8261c7 in term 2.
W20240501 19:56:12.671381 27059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240501 19:56:12.671607 27058 leader_election.cc:302] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b70afaac0df4baba858606cdf8261c7, e7369342d1c943a18c4de59daff48ca9; no voters:
I20240501 19:56:12.671856 27533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 19:56:12.671921 27533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:12.672010 27533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 2 FOLLOWER]: Advancing to term 3
W20240501 19:56:12.672073 27059 leader_election.cc:334] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329): Network error: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111)
I20240501 19:56:12.672995 27533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:12.673285 27533 leader_election.cc:288] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 3 election: Requested vote from peers ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329), e7369342d1c943a18c4de59daff48ca9 (127.25.83.60:36667)
I20240501 19:56:12.673573 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b70afaac0df4baba858606cdf8261c7" candidate_term: 3 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e7369342d1c943a18c4de59daff48ca9"
I20240501 19:56:12.673676 27148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 2 FOLLOWER]: Advancing to term 3
W20240501 19:56:12.674077 27059 leader_election.cc:334] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329): Network error: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111)
I20240501 19:56:12.674543 27148 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b70afaac0df4baba858606cdf8261c7 in term 3.
I20240501 19:56:12.676335 27058 leader_election.cc:302] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b70afaac0df4baba858606cdf8261c7, e7369342d1c943a18c4de59daff48ca9; no voters: ceb9e82c08394b9b8148930e35bd3533
I20240501 19:56:12.676704 27533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:56:12.676963 27533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 3 LEADER]: Becoming Leader. State: Replica: 8b70afaac0df4baba858606cdf8261c7, State: Running, Role: LEADER
I20240501 19:56:12.677150 27533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } }
I20240501 19:56:12.677937 27536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b70afaac0df4baba858606cdf8261c7. Latest consensus state: current_term: 3 leader_uuid: "8b70afaac0df4baba858606cdf8261c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:12.678092 27536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:12.678506 27538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:12.679857 27538 catalog_manager.cc:669] Loaded metadata for table test-workload [id=bba51bb011f644dea9102cd72be7e2db]
I20240501 19:56:12.680675 27538 tablet_loader.cc:96] loaded metadata for tablet 2550edc71e5545eaa3f154ed69764d6f (table test-workload [id=bba51bb011f644dea9102cd72be7e2db])
I20240501 19:56:12.680864 27538 tablet_loader.cc:96] loaded metadata for tablet ac11c808c56c4839b8549fc136c638b1 (table test-workload [id=bba51bb011f644dea9102cd72be7e2db])
I20240501 19:56:12.681059 27538 tablet_loader.cc:96] loaded metadata for tablet d15e5694a6eb4c20906892f761ebffcb (table test-workload [id=bba51bb011f644dea9102cd72be7e2db])
I20240501 19:56:12.681157 27538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:12.681546 27538 catalog_manager.cc:1259] Loaded cluster ID: 88fe7d5d274d4dc9905de8ac0315d480
I20240501 19:56:12.681592 27538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:12.682224 27538 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:12.682633 27538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: Loaded TSK: 0
I20240501 19:56:12.683207 27538 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20240501 19:56:13.083910 27059 consensus_peers.cc:485] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 -> Peer ceb9e82c08394b9b8148930e35bd3533 (127.25.83.62:45329): Couldn't send request to peer ceb9e82c08394b9b8148930e35bd3533. Status: Network error: Client connection negotiation failed: client connection to 127.25.83.62:45329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:13.085469 27214 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:13.085544 27214 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:13.179498 27148 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 3 FOLLOWER]: Refusing update from remote peer 8b70afaac0df4baba858606cdf8261c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 19:56:13.179977 27536 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:13.180882 27539 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b70afaac0df4baba858606cdf8261c7. Latest consensus state: current_term: 3 leader_uuid: "8b70afaac0df4baba858606cdf8261c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:13.180975 27539 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:13.181787 27536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8b70afaac0df4baba858606cdf8261c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:13.181870 27536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:13.182204 27539 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8b70afaac0df4baba858606cdf8261c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ceb9e82c08394b9b8148930e35bd3533" member_type: VOTER last_known_addr { host: "127.25.83.62" port: 45329 } } peers { permanent_uuid: "8b70afaac0df4baba858606cdf8261c7" member_type: VOTER last_known_addr { host: "127.25.83.61" port: 33593 } } peers { permanent_uuid: "e7369342d1c943a18c4de59daff48ca9" member_type: VOTER last_known_addr { host: "127.25.83.60" port: 36667 } } }
I20240501 19:56:13.182325 27539 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [sys.catalog]: This master's current role is: FOLLOWER
I20240501 19:56:13.245474 25932 tablet_server.cc:178] TabletServer@127.25.83.1:0 shutting down...
I20240501 19:56:13.249956 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:13.250664 25932 tablet_replica.cc:330] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c: stopping tablet replica
I20240501 19:56:13.250982 25932 raft_consensus.cc:2229] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:13.251258 25932 raft_consensus.cc:2258] T d15e5694a6eb4c20906892f761ebffcb P e05a078da6a84e0db2155dda659eae7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:13.264357 25932 tablet_server.cc:194] TabletServer@127.25.83.1:0 shutdown complete.
I20240501 19:56:13.266474 25932 tablet_server.cc:178] TabletServer@127.25.83.2:0 shutting down...
I20240501 19:56:13.270638 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:13.270864 25932 tablet_replica.cc:330] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a: stopping tablet replica
I20240501 19:56:13.271059 25932 raft_consensus.cc:2229] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:13.271211 25932 raft_consensus.cc:2258] T 2550edc71e5545eaa3f154ed69764d6f P 151e4fef4c614e559ac8dbbd249fc70a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:13.283416 25932 tablet_server.cc:194] TabletServer@127.25.83.2:0 shutdown complete.
I20240501 19:56:13.285271 25932 tablet_server.cc:178] TabletServer@127.25.83.3:0 shutting down...
I20240501 19:56:13.288743 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:13.288903 25932 tablet_replica.cc:330] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75: stopping tablet replica
I20240501 19:56:13.289011 25932 raft_consensus.cc:2229] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:13.289112 25932 raft_consensus.cc:2258] T ac11c808c56c4839b8549fc136c638b1 P b4e5d69334f14353a6a915fb7f1ebb75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:13.301273 25932 tablet_server.cc:194] TabletServer@127.25.83.3:0 shutdown complete.
I20240501 19:56:13.302989 25932 master.cc:537] Master@127.25.83.61:33593 shutting down...
I20240501 19:56:13.305815 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 3 LEADER]: Raft consensus shutting down.
I20240501 19:56:13.306061 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:13.306229 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8b70afaac0df4baba858606cdf8261c7: stopping tablet replica
I20240501 19:56:13.308650 25932 master.cc:552] Master@127.25.83.61:33593 shutdown complete.
I20240501 19:56:13.310787 25932 master.cc:537] Master@127.25.83.60:36667 shutting down...
I20240501 19:56:13.313792 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:13.313901 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:13.313956 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P e7369342d1c943a18c4de59daff48ca9: stopping tablet replica
I20240501 19:56:13.326004 25932 master.cc:552] Master@127.25.83.60:36667 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6394 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20240501 19:56:13.332881 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:34467
I20240501 19:56:13.333177 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:13.335626 27544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:13.336381 27545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:13.337121 27547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:13.338204 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:13.338375 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:13.338414 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:13.338429 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593373338429 us; error 0 us; skew 500 ppm
I20240501 19:56:13.339280 25932 webserver.cc:460] Webserver started at http://127.25.83.62:44881/ using document root <none> and password file <none>
I20240501 19:56:13.339412 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:13.339454 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:13.339511 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:13.339855 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "ae6e25da21a54a27896dfbba908d7a01"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-mfp3"
I20240501 19:56:13.341492 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:13.342424 27552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.342644 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:13.342705 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "ae6e25da21a54a27896dfbba908d7a01"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-mfp3"
I20240501 19:56:13.342767 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:13.370714 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:13.371107 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:13.375378 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:34467
I20240501 19:56:13.378576 27604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:34467 every 8 connection(s)
I20240501 19:56:13.379015 27605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:13.380813 27605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01: Bootstrap starting.
I20240501 19:56:13.381603 27605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:13.382638 27605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01: No bootstrap required, opened a new log
I20240501 19:56:13.382966 27605 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6e25da21a54a27896dfbba908d7a01" member_type: VOTER }
I20240501 19:56:13.383046 27605 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:13.383101 27605 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6e25da21a54a27896dfbba908d7a01, State: Initialized, Role: FOLLOWER
I20240501 19:56:13.383224 27605 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6e25da21a54a27896dfbba908d7a01" member_type: VOTER }
I20240501 19:56:13.383288 27605 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:13.383322 27605 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:13.383366 27605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:13.385600 27605 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6e25da21a54a27896dfbba908d7a01" member_type: VOTER }
I20240501 19:56:13.385740 27605 leader_election.cc:302] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6e25da21a54a27896dfbba908d7a01; no voters:
I20240501 19:56:13.385950 27605 leader_election.cc:288] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:13.386044 27608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:13.386281 27608 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 1 LEADER]: Becoming Leader. State: Replica: ae6e25da21a54a27896dfbba908d7a01, State: Running, Role: LEADER
I20240501 19:56:13.386404 27608 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6e25da21a54a27896dfbba908d7a01" member_type: VOTER }
I20240501 19:56:13.386519 27605 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:13.386754 27609 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6e25da21a54a27896dfbba908d7a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6e25da21a54a27896dfbba908d7a01" member_type: VOTER } }
I20240501 19:56:13.386754 27608 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6e25da21a54a27896dfbba908d7a01. Latest consensus state: current_term: 1 leader_uuid: "ae6e25da21a54a27896dfbba908d7a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6e25da21a54a27896dfbba908d7a01" member_type: VOTER } }
I20240501 19:56:13.386865 27609 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:13.386888 27608 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:13.387224 27612 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:13.388301 27612 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:13.388492 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:56:13.389894 27612 catalog_manager.cc:1347] Generated new cluster ID: cdf2b7528221438288ce771395d4af8e
I20240501 19:56:13.389948 27612 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:13.404316 27612 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:13.404815 27612 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:13.430953 27612 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01: Generated new TSK 0
I20240501 19:56:13.431145 27612 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:13.452986 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:13.455785 27626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:13.456755 27627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:13.457587 27629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:13.458405 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:13.458570 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:13.458608 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:13.458623 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593373458623 us; error 0 us; skew 500 ppm
I20240501 19:56:13.459483 25932 webserver.cc:460] Webserver started at http://127.25.83.1:41103/ using document root <none> and password file <none>
I20240501 19:56:13.459630 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:13.459678 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:13.459756 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:13.460168 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-0-root/instance:
uuid: "707b7eefa1da490f850b7982072172dc"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-mfp3"
I20240501 19:56:13.461593 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:13.462463 27634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.462714 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:13.462787 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-0-root
uuid: "707b7eefa1da490f850b7982072172dc"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-mfp3"
I20240501 19:56:13.462847 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:13.487565 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:13.487897 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:13.488148 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:13.488627 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:13.488665 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.488713 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:13.488742 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.492735 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.1:40945
I20240501 19:56:13.492769 27697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.1:40945 every 8 connection(s)
I20240501 19:56:13.493530 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:13.496385 27702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:13.497136 27703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:13.497869 27698 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34467
I20240501 19:56:13.497942 27698 heartbeater.cc:464] Registering TS with master...
W20240501 19:56:13.498142 27705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:13.498219 27698 heartbeater.cc:511] Master 127.25.83.62:34467 requested a full tablet report, sending...
I20240501 19:56:13.498765 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:13.498802 27569 ts_manager.cc:194] Registered new tserver with Master: 707b7eefa1da490f850b7982072172dc (127.25.83.1:40945)
I20240501 19:56:13.498943 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:13.498991 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:13.499011 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593373499011 us; error 0 us; skew 500 ppm
I20240501 19:56:13.499758 27569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41966
I20240501 19:56:13.499955 25932 webserver.cc:460] Webserver started at http://127.25.83.2:41209/ using document root <none> and password file <none>
I20240501 19:56:13.500115 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:13.500162 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:13.500212 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:13.500535 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-1-root/instance:
uuid: "9cd97e81fa4d468e8e71892d57662cdb"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-mfp3"
I20240501 19:56:13.501824 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:13.504220 27710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.504464 25932 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20240501 19:56:13.504530 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-1-root
uuid: "9cd97e81fa4d468e8e71892d57662cdb"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-mfp3"
I20240501 19:56:13.505148 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:13.518961 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:13.519240 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:13.519456 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:13.519850 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:13.519886 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.519913 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:13.519933 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.523687 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.2:46269
I20240501 19:56:13.523727 27773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.2:46269 every 8 connection(s)
I20240501 19:56:13.524348 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:13.527053 27778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:13.528021 27780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:13.528358 27774 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34467
I20240501 19:56:13.528436 27774 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:13.528620 27774 heartbeater.cc:511] Master 127.25.83.62:34467 requested a full tablet report, sending...
W20240501 19:56:13.528867 27777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:13.529170 27569 ts_manager.cc:194] Registered new tserver with Master: 9cd97e81fa4d468e8e71892d57662cdb (127.25.83.2:46269)
I20240501 19:56:13.529541 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:13.529686 27569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41976
I20240501 19:56:13.529724 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:13.529762 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:13.529796 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593373529795 us; error 0 us; skew 500 ppm
I20240501 19:56:13.530689 25932 webserver.cc:460] Webserver started at http://127.25.83.3:43217/ using document root <none> and password file <none>
I20240501 19:56:13.530827 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:13.530874 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:13.530941 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:13.531272 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-2-root/instance:
uuid: "ccc6101940484878be8c6d1455fc7f8d"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-mfp3"
I20240501 19:56:13.532558 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:13.533442 27785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.533643 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:13.533705 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-2-root
uuid: "ccc6101940484878be8c6d1455fc7f8d"
format_stamp: "Formatted at 2024-05-01 19:56:13 on dist-test-slave-mfp3"
I20240501 19:56:13.533763 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:13.544162 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:13.544448 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:13.544677 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:13.545097 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:13.545135 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.545181 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:13.545207 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.552443 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.3:35053
I20240501 19:56:13.552450 27848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.3:35053 every 8 connection(s)
I20240501 19:56:13.557164 27849 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34467
I20240501 19:56:13.557267 27849 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:13.557471 27849 heartbeater.cc:511] Master 127.25.83.62:34467 requested a full tablet report, sending...
I20240501 19:56:13.558033 27569 ts_manager.cc:194] Registered new tserver with Master: ccc6101940484878be8c6d1455fc7f8d (127.25.83.3:35053)
I20240501 19:56:13.558658 27569 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41982
I20240501 19:56:13.558705 25932 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005746046s
I20240501 19:56:14.500738 27698 heartbeater.cc:503] Master 127.25.83.62:34467 was elected leader, sending a full tablet report...
I20240501 19:56:14.530544 27774 heartbeater.cc:503] Master 127.25.83.62:34467 was elected leader, sending a full tablet report...
I20240501 19:56:14.559545 27849 heartbeater.cc:503] Master 127.25.83.62:34467 was elected leader, sending a full tablet report...
I20240501 19:56:14.584956 25932 test_util.cc:268] Using random seed: -519651207
I20240501 19:56:14.592388 27569 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41992:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:56:14.592792 27569 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:56:14.601603 27738 tablet_service.cc:1467] Processing CreateTablet for tablet c1ccd4b16f9f4b2caffdfb3d9b547cd1 (DEFAULT_TABLE table=test-workload [id=559d21d398524400818232ccbd0b6799]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 19:56:14.601632 27737 tablet_service.cc:1467] Processing CreateTablet for tablet 253365737b4c40f8a7684abb8b807fae (DEFAULT_TABLE table=test-workload [id=559d21d398524400818232ccbd0b6799]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 19:56:14.601920 27738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ccd4b16f9f4b2caffdfb3d9b547cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:14.602174 27662 tablet_service.cc:1467] Processing CreateTablet for tablet c1ccd4b16f9f4b2caffdfb3d9b547cd1 (DEFAULT_TABLE table=test-workload [id=559d21d398524400818232ccbd0b6799]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 19:56:14.602237 27661 tablet_service.cc:1467] Processing CreateTablet for tablet 253365737b4c40f8a7684abb8b807fae (DEFAULT_TABLE table=test-workload [id=559d21d398524400818232ccbd0b6799]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 19:56:14.602416 27662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ccd4b16f9f4b2caffdfb3d9b547cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:14.602439 27813 tablet_service.cc:1467] Processing CreateTablet for tablet c1ccd4b16f9f4b2caffdfb3d9b547cd1 (DEFAULT_TABLE table=test-workload [id=559d21d398524400818232ccbd0b6799]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20240501 19:56:14.602566 27661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253365737b4c40f8a7684abb8b807fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:14.602644 27813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1ccd4b16f9f4b2caffdfb3d9b547cd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:14.604460 27869 tablet_bootstrap.cc:492] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc: Bootstrap starting.
I20240501 19:56:14.604692 27811 tablet_service.cc:1467] Processing CreateTablet for tablet 253365737b4c40f8a7684abb8b807fae (DEFAULT_TABLE table=test-workload [id=559d21d398524400818232ccbd0b6799]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20240501 19:56:14.604768 27870 tablet_bootstrap.cc:492] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d: Bootstrap starting.
I20240501 19:56:14.604916 27811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253365737b4c40f8a7684abb8b807fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:14.605453 27869 tablet_bootstrap.cc:654] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:14.605856 27870 tablet_bootstrap.cc:654] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:14.606549 27869 tablet_bootstrap.cc:492] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc: No bootstrap required, opened a new log
I20240501 19:56:14.606685 27869 ts_tablet_manager.cc:1367] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:14.606835 27872 tablet_bootstrap.cc:492] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb: Bootstrap starting.
I20240501 19:56:14.607013 27737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253365737b4c40f8a7684abb8b807fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:14.607126 27869 raft_consensus.cc:348] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.607228 27869 raft_consensus.cc:374] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:14.607259 27869 raft_consensus.cc:729] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707b7eefa1da490f850b7982072172dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:14.607424 27869 consensus_queue.cc:260] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.607764 27869 ts_tablet_manager.cc:1398] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:14.607915 27869 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc: Bootstrap starting.
I20240501 19:56:14.608812 27869 tablet_bootstrap.cc:654] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:14.608979 27872 tablet_bootstrap.cc:654] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:14.609937 27872 tablet_bootstrap.cc:492] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb: No bootstrap required, opened a new log
I20240501 19:56:14.609937 27870 tablet_bootstrap.cc:492] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d: No bootstrap required, opened a new log
I20240501 19:56:14.610100 27870 ts_tablet_manager.cc:1367] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:14.610272 27869 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc: No bootstrap required, opened a new log
I20240501 19:56:14.610350 27869 ts_tablet_manager.cc:1367] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:14.610493 27870 raft_consensus.cc:348] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.610574 27870 raft_consensus.cc:374] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:14.610596 27870 raft_consensus.cc:729] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc6101940484878be8c6d1455fc7f8d, State: Initialized, Role: FOLLOWER
I20240501 19:56:14.610702 27869 raft_consensus.cc:348] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.610765 27872 ts_tablet_manager.cc:1367] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:14.610788 27869 raft_consensus.cc:374] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:14.610713 27870 consensus_queue.cc:260] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.610857 27869 raft_consensus.cc:729] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707b7eefa1da490f850b7982072172dc, State: Initialized, Role: FOLLOWER
I20240501 19:56:14.610998 27869 consensus_queue.cc:260] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.611102 27870 ts_tablet_manager.cc:1398] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:14.611266 27870 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: Bootstrap starting.
I20240501 19:56:14.611474 27872 raft_consensus.cc:348] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.611557 27872 raft_consensus.cc:374] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:14.611584 27872 raft_consensus.cc:729] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd97e81fa4d468e8e71892d57662cdb, State: Initialized, Role: FOLLOWER
I20240501 19:56:14.611701 27872 consensus_queue.cc:260] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.611981 27872 ts_tablet_manager.cc:1398] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:14.612145 27870 tablet_bootstrap.cc:654] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:14.612181 27872 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb: Bootstrap starting.
I20240501 19:56:14.612857 27869 ts_tablet_manager.cc:1398] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:14.613008 27872 tablet_bootstrap.cc:654] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:14.613932 27872 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb: No bootstrap required, opened a new log
I20240501 19:56:14.614013 27872 ts_tablet_manager.cc:1367] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:14.614456 27872 raft_consensus.cc:348] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.614560 27872 raft_consensus.cc:374] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:14.614624 27872 raft_consensus.cc:729] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd97e81fa4d468e8e71892d57662cdb, State: Initialized, Role: FOLLOWER
I20240501 19:56:14.614742 27872 consensus_queue.cc:260] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.614908 27870 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: No bootstrap required, opened a new log
I20240501 19:56:14.614992 27870 ts_tablet_manager.cc:1367] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:14.615069 27872 ts_tablet_manager.cc:1398] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:14.615383 27870 raft_consensus.cc:348] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.615497 27870 raft_consensus.cc:374] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:14.615538 27870 raft_consensus.cc:729] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccc6101940484878be8c6d1455fc7f8d, State: Initialized, Role: FOLLOWER
I20240501 19:56:14.615659 27870 consensus_queue.cc:260] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.615893 27870 ts_tablet_manager.cc:1398] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:14.676278 27878 raft_consensus.cc:482] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:14.676440 27878 raft_consensus.cc:504] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.677038 27878 leader_election.cc:288] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 707b7eefa1da490f850b7982072172dc (127.25.83.1:40945), ccc6101940484878be8c6d1455fc7f8d (127.25.83.3:35053)
I20240501 19:56:14.681216 27672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253365737b4c40f8a7684abb8b807fae" candidate_uuid: "9cd97e81fa4d468e8e71892d57662cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "707b7eefa1da490f850b7982072172dc" is_pre_election: true
I20240501 19:56:14.681290 27823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253365737b4c40f8a7684abb8b807fae" candidate_uuid: "9cd97e81fa4d468e8e71892d57662cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc6101940484878be8c6d1455fc7f8d" is_pre_election: true
I20240501 19:56:14.681362 27672 raft_consensus.cc:2454] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd97e81fa4d468e8e71892d57662cdb in term 0.
I20240501 19:56:14.681406 27823 raft_consensus.cc:2454] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd97e81fa4d468e8e71892d57662cdb in term 0.
I20240501 19:56:14.681635 27713 leader_election.cc:302] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707b7eefa1da490f850b7982072172dc, 9cd97e81fa4d468e8e71892d57662cdb; no voters:
I20240501 19:56:14.681777 27878 raft_consensus.cc:2781] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:14.681836 27878 raft_consensus.cc:482] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:14.681885 27878 raft_consensus.cc:3037] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:14.682724 27878 raft_consensus.cc:504] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.683035 27878 leader_election.cc:288] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [CANDIDATE]: Term 1 election: Requested vote from peers 707b7eefa1da490f850b7982072172dc (127.25.83.1:40945), ccc6101940484878be8c6d1455fc7f8d (127.25.83.3:35053)
I20240501 19:56:14.683271 27672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253365737b4c40f8a7684abb8b807fae" candidate_uuid: "9cd97e81fa4d468e8e71892d57662cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "707b7eefa1da490f850b7982072172dc"
I20240501 19:56:14.683343 27823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "253365737b4c40f8a7684abb8b807fae" candidate_uuid: "9cd97e81fa4d468e8e71892d57662cdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccc6101940484878be8c6d1455fc7f8d"
I20240501 19:56:14.683370 27672 raft_consensus.cc:3037] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:14.683445 27823 raft_consensus.cc:3037] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:14.684090 27672 raft_consensus.cc:2454] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd97e81fa4d468e8e71892d57662cdb in term 1.
I20240501 19:56:14.684253 27823 raft_consensus.cc:2454] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd97e81fa4d468e8e71892d57662cdb in term 1.
I20240501 19:56:14.684329 27713 leader_election.cc:302] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707b7eefa1da490f850b7982072172dc, 9cd97e81fa4d468e8e71892d57662cdb; no voters:
I20240501 19:56:14.684484 27878 raft_consensus.cc:2781] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:14.684670 27878 raft_consensus.cc:686] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 1 LEADER]: Becoming Leader. State: Replica: 9cd97e81fa4d468e8e71892d57662cdb, State: Running, Role: LEADER
I20240501 19:56:14.684814 27878 consensus_queue.cc:237] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.686228 27567 catalog_manager.cc:5484] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd97e81fa4d468e8e71892d57662cdb (127.25.83.2). New cstate: current_term: 1 leader_uuid: "9cd97e81fa4d468e8e71892d57662cdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:14.753888 27877 raft_consensus.cc:482] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:14.754022 27877 raft_consensus.cc:504] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.754496 27877 leader_election.cc:288] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cd97e81fa4d468e8e71892d57662cdb (127.25.83.2:46269), 707b7eefa1da490f850b7982072172dc (127.25.83.1:40945)
I20240501 19:56:14.757908 27672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ccd4b16f9f4b2caffdfb3d9b547cd1" candidate_uuid: "ccc6101940484878be8c6d1455fc7f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "707b7eefa1da490f850b7982072172dc" is_pre_election: true
I20240501 19:56:14.757979 27748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ccd4b16f9f4b2caffdfb3d9b547cd1" candidate_uuid: "ccc6101940484878be8c6d1455fc7f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd97e81fa4d468e8e71892d57662cdb" is_pre_election: true
I20240501 19:56:14.758203 27672 raft_consensus.cc:2454] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc6101940484878be8c6d1455fc7f8d in term 0.
I20240501 19:56:14.758242 27748 raft_consensus.cc:2454] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccc6101940484878be8c6d1455fc7f8d in term 0.
I20240501 19:56:14.758502 27788 leader_election.cc:302] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707b7eefa1da490f850b7982072172dc, ccc6101940484878be8c6d1455fc7f8d; no voters:
I20240501 19:56:14.758668 27877 raft_consensus.cc:2781] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:14.758728 27877 raft_consensus.cc:482] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:14.758767 27877 raft_consensus.cc:3037] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:14.760429 27877 raft_consensus.cc:504] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.760761 27877 leader_election.cc:288] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [CANDIDATE]: Term 1 election: Requested vote from peers 9cd97e81fa4d468e8e71892d57662cdb (127.25.83.2:46269), 707b7eefa1da490f850b7982072172dc (127.25.83.1:40945)
I20240501 19:56:14.761090 27672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ccd4b16f9f4b2caffdfb3d9b547cd1" candidate_uuid: "ccc6101940484878be8c6d1455fc7f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "707b7eefa1da490f850b7982072172dc"
I20240501 19:56:14.761090 27748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1ccd4b16f9f4b2caffdfb3d9b547cd1" candidate_uuid: "ccc6101940484878be8c6d1455fc7f8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd97e81fa4d468e8e71892d57662cdb"
I20240501 19:56:14.761205 27672 raft_consensus.cc:3037] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:14.761216 27748 raft_consensus.cc:3037] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:14.762039 27748 raft_consensus.cc:2454] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc6101940484878be8c6d1455fc7f8d in term 1.
I20240501 19:56:14.762028 27672 raft_consensus.cc:2454] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccc6101940484878be8c6d1455fc7f8d in term 1.
I20240501 19:56:14.762382 27788 leader_election.cc:302] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707b7eefa1da490f850b7982072172dc, ccc6101940484878be8c6d1455fc7f8d; no voters:
I20240501 19:56:14.762524 27877 raft_consensus.cc:2781] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:14.762737 27877 raft_consensus.cc:686] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 1 LEADER]: Becoming Leader. State: Replica: ccc6101940484878be8c6d1455fc7f8d, State: Running, Role: LEADER
I20240501 19:56:14.762908 27877 consensus_queue.cc:237] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } }
I20240501 19:56:14.764427 27567 catalog_manager.cc:5484] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d reported cstate change: term changed from 0 to 1, leader changed from <none> to ccc6101940484878be8c6d1455fc7f8d (127.25.83.3). New cstate: current_term: 1 leader_uuid: "ccc6101940484878be8c6d1455fc7f8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:15.136682 27878 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:15.139508 27883 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:15.169260 27877 consensus_queue.cc:1035] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:15.171846 27877 consensus_queue.cc:1035] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:15.388664 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:15.389214 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:15.389314 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:17.389609 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:17.390048 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:17.390131 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:19.390475 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:19.390919 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:19.390986 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:21.391259 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:21.391671 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:21.391728 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:23.392015 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:23.392436 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:23.392496 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:25.392776 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:25.393201 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:25.393263 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:25.840268 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:25.843364 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:25.844069 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:25.845075 25932 server_base.cc:1034] running on GCE node
W20240501 19:56:25.845899 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:25.846160 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:25.846217 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:25.846253 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593385846252 us; error 0 us; skew 500 ppm
I20240501 19:56:25.847352 25932 webserver.cc:460] Webserver started at http://127.25.83.4:41923/ using document root <none> and password file <none>
I20240501 19:56:25.847512 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:25.847571 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:25.847654 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:25.848057 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-3-root/instance:
uuid: "6bcfc39db20241c6a94206eee408dd6f"
format_stamp: "Formatted at 2024-05-01 19:56:25 on dist-test-slave-mfp3"
I20240501 19:56:25.849511 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:25.850474 27913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:25.850771 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.850833 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-3-root
uuid: "6bcfc39db20241c6a94206eee408dd6f"
format_stamp: "Formatted at 2024-05-01 19:56:25 on dist-test-slave-mfp3"
I20240501 19:56:25.850891 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:25.861199 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:25.861894 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:25.862257 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:25.862793 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:25.862834 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:25.862866 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:25.862880 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:25.868180 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.4:37303
I20240501 19:56:25.868292 27976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.4:37303 every 8 connection(s)
I20240501 19:56:25.873203 27977 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34467
I20240501 19:56:25.873296 27977 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:25.873484 27977 heartbeater.cc:511] Master 127.25.83.62:34467 requested a full tablet report, sending...
I20240501 19:56:25.874009 27567 ts_manager.cc:194] Registered new tserver with Master: 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4:37303)
I20240501 19:56:25.874555 27567 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37762
I20240501 19:56:26.402660 27748 consensus_queue.cc:237] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: true } }
I20240501 19:56:26.403949 27823 raft_consensus.cc:1261] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Refusing update from remote peer 9cd97e81fa4d468e8e71892d57662cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:26.403978 27672 raft_consensus.cc:1261] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Refusing update from remote peer 9cd97e81fa4d468e8e71892d57662cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:26.404347 27985 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:26.404544 27986 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240501 19:56:26.406075 27985 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: true } } }
I20240501 19:56:26.406625 27672 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: true } } }
W20240501 19:56:26.407701 27712 consensus_peers.cc:485] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb -> Peer 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4:37303): Couldn't send request to peer 6bcfc39db20241c6a94206eee408dd6f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 253365737b4c40f8a7684abb8b807fae. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:26.408274 27823 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: true } } }
I20240501 19:56:26.409179 27567 catalog_manager.cc:5484] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb reported cstate change: config changed from index -1 to 2, NON_VOTER 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) added. New cstate: current_term: 1 leader_uuid: "9cd97e81fa4d468e8e71892d57662cdb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:26.875283 27977 heartbeater.cc:503] Master 127.25.83.62:34467 was elected leader, sending a full tablet report...
I20240501 19:56:27.001091 27996 ts_tablet_manager.cc:902] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: Initiating tablet copy from peer 9cd97e81fa4d468e8e71892d57662cdb (127.25.83.2:46269)
I20240501 19:56:27.001832 27996 tablet_copy_client.cc:308] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.2:46269
I20240501 19:56:27.005553 27758 tablet_copy_service.cc:140] P 9cd97e81fa4d468e8e71892d57662cdb: Received BeginTabletCopySession request for tablet 253365737b4c40f8a7684abb8b807fae from peer 6bcfc39db20241c6a94206eee408dd6f ({username='slave'} at 127.0.0.1:55516)
I20240501 19:56:27.005664 27758 tablet_copy_service.cc:161] P 9cd97e81fa4d468e8e71892d57662cdb: Beginning new tablet copy session on tablet 253365737b4c40f8a7684abb8b807fae from peer 6bcfc39db20241c6a94206eee408dd6f at {username='slave'} at 127.0.0.1:55516: session id = 6bcfc39db20241c6a94206eee408dd6f-253365737b4c40f8a7684abb8b807fae
I20240501 19:56:27.007146 27758 tablet_copy_source_session.cc:205] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:27.008450 27996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 253365737b4c40f8a7684abb8b807fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.010419 27996 tablet_copy_client.cc:783] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:27.010576 27996 tablet_copy_client.cc:647] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:27.011771 27996 tablet_copy_client.cc:523] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:27.013175 27996 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: Bootstrap starting.
I20240501 19:56:27.017243 27996 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:27.017495 27996 tablet_bootstrap.cc:492] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: Bootstrap complete.
I20240501 19:56:27.017648 27996 ts_tablet_manager.cc:1367] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:27.018034 27996 raft_consensus.cc:348] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: true } }
I20240501 19:56:27.018167 27996 raft_consensus.cc:729] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bcfc39db20241c6a94206eee408dd6f, State: Initialized, Role: LEARNER
I20240501 19:56:27.018268 27996 consensus_queue.cc:260] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: true } }
I20240501 19:56:27.018595 27996 ts_tablet_manager.cc:1398] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:27.019641 27758 tablet_copy_service.cc:342] P 9cd97e81fa4d468e8e71892d57662cdb: Request end of tablet copy session 6bcfc39db20241c6a94206eee408dd6f-253365737b4c40f8a7684abb8b807fae received from {username='slave'} at 127.0.0.1:55516
I20240501 19:56:27.019711 27758 tablet_copy_service.cc:433] P 9cd97e81fa4d468e8e71892d57662cdb: ending tablet copy session 6bcfc39db20241c6a94206eee408dd6f-253365737b4c40f8a7684abb8b807fae on tablet 253365737b4c40f8a7684abb8b807fae with peer 6bcfc39db20241c6a94206eee408dd6f
I20240501 19:56:27.393484 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:27.393846 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:27.393923 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:27.470937 27951 raft_consensus.cc:1203] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240501 19:56:27.908075 27981 raft_consensus.cc:1050] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb: attempting to promote NON_VOTER 6bcfc39db20241c6a94206eee408dd6f to VOTER
I20240501 19:56:27.908656 27981 consensus_queue.cc:237] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } }
I20240501 19:56:27.909857 27951 raft_consensus.cc:1261] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 LEARNER]: Refusing update from remote peer 9cd97e81fa4d468e8e71892d57662cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:27.909890 27823 raft_consensus.cc:1261] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Refusing update from remote peer 9cd97e81fa4d468e8e71892d57662cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:27.910041 27672 raft_consensus.cc:1261] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Refusing update from remote peer 9cd97e81fa4d468e8e71892d57662cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:56:27.910265 28006 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:27.910440 28005 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:27.910773 28005 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:27.912346 28006 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } } }
I20240501 19:56:27.912716 27823 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } } }
I20240501 19:56:27.913856 27951 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } } }
I20240501 19:56:27.914388 27672 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } } }
I20240501 19:56:27.914522 27567 catalog_manager.cc:5484] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d reported cstate change: config changed from index 2 to 3, 6bcfc39db20241c6a94206eee408dd6f (127.25.83.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9cd97e81fa4d468e8e71892d57662cdb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "ccc6101940484878be8c6d1455fc7f8d" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 35053 } attrs { replace: true } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } } }
I20240501 19:56:27.919343 27748 consensus_queue.cc:237] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } }
I20240501 19:56:27.920274 27951 raft_consensus.cc:1261] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 FOLLOWER]: Refusing update from remote peer 9cd97e81fa4d468e8e71892d57662cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:27.920369 27672 raft_consensus.cc:1261] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Refusing update from remote peer 9cd97e81fa4d468e8e71892d57662cdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:27.920568 28008 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:27.920713 28005 consensus_queue.cc:1035] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:27.922041 28006 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ccc6101940484878be8c6d1455fc7f8d (127.25.83.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } } }
I20240501 19:56:27.922415 27672 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ccc6101940484878be8c6d1455fc7f8d (127.25.83.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } } }
I20240501 19:56:27.923622 27951 raft_consensus.cc:2932] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ccc6101940484878be8c6d1455fc7f8d (127.25.83.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } } }
I20240501 19:56:27.923764 27553 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 253365737b4c40f8a7684abb8b807fae with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:27.924211 27568 catalog_manager.cc:5484] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb reported cstate change: config changed from index 3 to 4, VOTER ccc6101940484878be8c6d1455fc7f8d (127.25.83.3) evicted. New cstate: current_term: 1 leader_uuid: "9cd97e81fa4d468e8e71892d57662cdb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "707b7eefa1da490f850b7982072172dc" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 40945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9cd97e81fa4d468e8e71892d57662cdb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 46269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bcfc39db20241c6a94206eee408dd6f" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 37303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:27.926368 27811 tablet_service.cc:1514] Processing DeleteTablet for tablet 253365737b4c40f8a7684abb8b807fae with delete_type TABLET_DATA_TOMBSTONED (TS ccc6101940484878be8c6d1455fc7f8d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36748
I20240501 19:56:27.926749 28017 tablet_replica.cc:330] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: stopping tablet replica
I20240501 19:56:27.926865 28017 raft_consensus.cc:2229] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:27.926992 28017 raft_consensus.cc:2258] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:27.927554 28017 ts_tablet_manager.cc:1875] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:27.929450 28017 ts_tablet_manager.cc:1888] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240501 19:56:27.929522 28017 log.cc:1196] T 253365737b4c40f8a7684abb8b807fae P ccc6101940484878be8c6d1455fc7f8d: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1714593357002142-25932-0/minicluster-data/ts-2-root/wals/253365737b4c40f8a7684abb8b807fae
I20240501 19:56:27.930083 27556 catalog_manager.cc:4830] TS ccc6101940484878be8c6d1455fc7f8d (127.25.83.3:35053): tablet 253365737b4c40f8a7684abb8b807fae (table test-workload [id=559d21d398524400818232ccbd0b6799]) successfully deleted
I20240501 19:56:29.394270 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:29.394675 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:29.394748 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:31.395066 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:31.395464 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:31.395529 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:33.395834 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:33.396195 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:33.396270 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:35.396695 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:35.397138 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:35.397205 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:37.397579 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:37.398010 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:37.398073 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:39.398411 27623 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:56:39.398808 27623 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:56:39.398869 27623 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:40.074488 25932 tablet_server.cc:178] TabletServer@127.25.83.1:0 shutting down...
I20240501 19:56:40.077962 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:40.078172 25932 tablet_replica.cc:330] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc: stopping tablet replica
I20240501 19:56:40.078294 25932 raft_consensus.cc:2229] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.078469 25932 raft_consensus.cc:2258] T 253365737b4c40f8a7684abb8b807fae P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.078923 25932 tablet_replica.cc:330] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc: stopping tablet replica
I20240501 19:56:40.078997 25932 raft_consensus.cc:2229] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.079092 25932 raft_consensus.cc:2258] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 707b7eefa1da490f850b7982072172dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.091573 25932 tablet_server.cc:194] TabletServer@127.25.83.1:0 shutdown complete.
I20240501 19:56:40.093443 25932 tablet_server.cc:178] TabletServer@127.25.83.2:0 shutting down...
I20240501 19:56:40.096885 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:40.097061 25932 tablet_replica.cc:330] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb: stopping tablet replica
I20240501 19:56:40.097151 25932 raft_consensus.cc:2229] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:40.097321 25932 raft_consensus.cc:2258] T 253365737b4c40f8a7684abb8b807fae P 9cd97e81fa4d468e8e71892d57662cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.097656 25932 tablet_replica.cc:330] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb: stopping tablet replica
I20240501 19:56:40.097733 25932 raft_consensus.cc:2229] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.097818 25932 raft_consensus.cc:2258] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P 9cd97e81fa4d468e8e71892d57662cdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.110107 25932 tablet_server.cc:194] TabletServer@127.25.83.2:0 shutdown complete.
I20240501 19:56:40.111804 25932 tablet_server.cc:178] TabletServer@127.25.83.3:0 shutting down...
I20240501 19:56:40.115583 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:40.115739 25932 tablet_replica.cc:330] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d: stopping tablet replica
I20240501 19:56:40.115811 25932 raft_consensus.cc:2229] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:40.115949 25932 raft_consensus.cc:2258] T c1ccd4b16f9f4b2caffdfb3d9b547cd1 P ccc6101940484878be8c6d1455fc7f8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.127898 25932 tablet_server.cc:194] TabletServer@127.25.83.3:0 shutdown complete.
I20240501 19:56:40.129422 25932 tablet_server.cc:178] TabletServer@127.25.83.4:0 shutting down...
I20240501 19:56:40.132853 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:56:40.132984 25932 tablet_replica.cc:330] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f: stopping tablet replica
I20240501 19:56:40.133060 25932 raft_consensus.cc:2229] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:40.133147 25932 raft_consensus.cc:2258] T 253365737b4c40f8a7684abb8b807fae P 6bcfc39db20241c6a94206eee408dd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.145359 25932 tablet_server.cc:194] TabletServer@127.25.83.4:0 shutdown complete.
I20240501 19:56:40.147161 25932 master.cc:537] Master@127.25.83.62:34467 shutting down...
I20240501 19:56:40.150310 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:40.150453 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:40.150504 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P ae6e25da21a54a27896dfbba908d7a01: stopping tablet replica
I20240501 19:56:40.162840 25932 master.cc:552] Master@127.25.83.62:34467 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26837 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20240501 19:56:40.170255 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:39943
I20240501 19:56:40.170557 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:40.172859 28021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:40.173396 28023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.173946 25932 server_base.cc:1034] running on GCE node
W20240501 19:56:40.174583 28020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:40.174782 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:40.174826 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:40.174847 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593400174847 us; error 0 us; skew 500 ppm
I20240501 19:56:40.179602 25932 webserver.cc:460] Webserver started at http://127.25.83.62:41881/ using document root <none> and password file <none>
I20240501 19:56:40.179747 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:40.179795 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:40.179869 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:40.180217 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "ef63e1b7e0b04680aaa803f18a15f22b"
format_stamp: "Formatted at 2024-05-01 19:56:40 on dist-test-slave-mfp3"
I20240501 19:56:40.181492 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.182330 28028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:40.182547 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:40.182610 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "ef63e1b7e0b04680aaa803f18a15f22b"
format_stamp: "Formatted at 2024-05-01 19:56:40 on dist-test-slave-mfp3"
I20240501 19:56:40.182682 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:40.200008 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:40.200291 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:40.203953 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:39943
I20240501 19:56:40.205999 28080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:39943 every 8 connection(s)
I20240501 19:56:40.206421 28081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:40.207927 28081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b: Bootstrap starting.
I20240501 19:56:40.208586 28081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:40.209432 28081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b: No bootstrap required, opened a new log
I20240501 19:56:40.209761 28081 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63e1b7e0b04680aaa803f18a15f22b" member_type: VOTER }
I20240501 19:56:40.209831 28081 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:40.209851 28081 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef63e1b7e0b04680aaa803f18a15f22b, State: Initialized, Role: FOLLOWER
I20240501 19:56:40.209931 28081 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63e1b7e0b04680aaa803f18a15f22b" member_type: VOTER }
I20240501 19:56:40.209976 28081 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:40.209995 28081 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:40.210023 28081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:40.210552 28081 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63e1b7e0b04680aaa803f18a15f22b" member_type: VOTER }
I20240501 19:56:40.210654 28081 leader_election.cc:302] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef63e1b7e0b04680aaa803f18a15f22b; no voters:
I20240501 19:56:40.210774 28081 leader_election.cc:288] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:40.210862 28084 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:40.211031 28084 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 1 LEADER]: Becoming Leader. State: Replica: ef63e1b7e0b04680aaa803f18a15f22b, State: Running, Role: LEADER
I20240501 19:56:40.211136 28081 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:40.211138 28084 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63e1b7e0b04680aaa803f18a15f22b" member_type: VOTER }
I20240501 19:56:40.211527 28085 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef63e1b7e0b04680aaa803f18a15f22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63e1b7e0b04680aaa803f18a15f22b" member_type: VOTER } }
I20240501 19:56:40.211633 28085 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.211539 28086 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef63e1b7e0b04680aaa803f18a15f22b. Latest consensus state: current_term: 1 leader_uuid: "ef63e1b7e0b04680aaa803f18a15f22b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef63e1b7e0b04680aaa803f18a15f22b" member_type: VOTER } }
I20240501 19:56:40.211882 28086 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:40.211901 28089 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:40.212927 28089 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:40.213099 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:56:40.214447 28089 catalog_manager.cc:1347] Generated new cluster ID: 11a38b60259040bebe969a5e1d650d66
I20240501 19:56:40.214493 28089 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:40.231050 28089 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:40.231484 28089 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:40.235738 28089 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b: Generated new TSK 0
I20240501 19:56:40.235862 28089 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:40.244854 25932 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 4.6259e-05s
I20240501 19:56:52.323010 25932 master.cc:537] Master@127.25.83.62:39943 shutting down...
I20240501 19:56:52.325855 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:56:52.326001 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.326061 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P ef63e1b7e0b04680aaa803f18a15f22b: stopping tablet replica
I20240501 19:56:52.339133 25932 master.cc:552] Master@127.25.83.62:39943 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12172 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20240501 19:56:52.343194 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:39855
I20240501 19:56:52.343466 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:56:52.345927 25932 server_base.cc:1034] running on GCE node
W20240501 19:56:52.346474 28104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.347291 28103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.348182 28106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.348728 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:52.348838 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:52.348870 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593412348869 us; error 0 us; skew 500 ppm
I20240501 19:56:52.354630 25932 webserver.cc:460] Webserver started at http://127.25.83.62:37663/ using document root <none> and password file <none>
I20240501 19:56:52.354825 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.354884 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.354967 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.355577 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "7ad7bdb64747416a9eb0060ee4d838fb"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-mfp3"
I20240501 19:56:52.357331 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:52.358389 28111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.358637 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.358698 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "7ad7bdb64747416a9eb0060ee4d838fb"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-mfp3"
I20240501 19:56:52.358753 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.369736 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.370069 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.373792 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:39855
I20240501 19:56:52.374447 28163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:39855 every 8 connection(s)
I20240501 19:56:52.374853 28164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.376478 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb: Bootstrap starting.
I20240501 19:56:52.377133 28164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:52.378077 28164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb: No bootstrap required, opened a new log
I20240501 19:56:52.378407 28164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad7bdb64747416a9eb0060ee4d838fb" member_type: VOTER }
I20240501 19:56:52.378489 28164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:52.378509 28164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad7bdb64747416a9eb0060ee4d838fb, State: Initialized, Role: FOLLOWER
I20240501 19:56:52.378609 28164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad7bdb64747416a9eb0060ee4d838fb" member_type: VOTER }
I20240501 19:56:52.378683 28164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:52.378713 28164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:52.378746 28164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:52.379314 28164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad7bdb64747416a9eb0060ee4d838fb" member_type: VOTER }
I20240501 19:56:52.379412 28164 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad7bdb64747416a9eb0060ee4d838fb; no voters:
I20240501 19:56:52.379539 28164 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:52.379647 28167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:52.379817 28167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 1 LEADER]: Becoming Leader. State: Replica: 7ad7bdb64747416a9eb0060ee4d838fb, State: Running, Role: LEADER
I20240501 19:56:52.379971 28164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:52.379958 28167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad7bdb64747416a9eb0060ee4d838fb" member_type: VOTER }
I20240501 19:56:52.380367 28168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad7bdb64747416a9eb0060ee4d838fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad7bdb64747416a9eb0060ee4d838fb" member_type: VOTER } }
I20240501 19:56:52.380441 28168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:52.380640 28169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad7bdb64747416a9eb0060ee4d838fb. Latest consensus state: current_term: 1 leader_uuid: "7ad7bdb64747416a9eb0060ee4d838fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad7bdb64747416a9eb0060ee4d838fb" member_type: VOTER } }
I20240501 19:56:52.380697 28172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:52.380782 28169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:52.381516 28172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:52.381765 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:56:52.383145 28172 catalog_manager.cc:1347] Generated new cluster ID: 103e884d90ba40cc8d11febf6ec53120
I20240501 19:56:52.383194 28172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:52.392062 28172 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:52.392472 28172 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:52.397184 28172 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb: Generated new TSK 0
I20240501 19:56:52.397313 28172 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:52.413961 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:52.416584 28186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.417053 28188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.417568 28185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.418269 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:52.418423 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:52.418479 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:52.418505 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593412418504 us; error 0 us; skew 500 ppm
I20240501 19:56:52.419375 25932 webserver.cc:460] Webserver started at http://127.25.83.1:35705/ using document root <none> and password file <none>
I20240501 19:56:52.419507 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.419564 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.419637 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.419975 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-0-root/instance:
uuid: "dcce477835bb469d90f6914314c22286"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-mfp3"
I20240501 19:56:52.421211 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.421897 28193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.422077 25932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 19:56:52.422152 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-0-root
uuid: "dcce477835bb469d90f6914314c22286"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-mfp3"
I20240501 19:56:52.422207 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.440177 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.440466 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.440733 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:52.441146 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:52.441182 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.441226 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:52.441263 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.445179 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.1:45151
I20240501 19:56:52.445206 28256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.1:45151 every 8 connection(s)
I20240501 19:56:52.445950 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:52.448473 28262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.449208 28264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.449698 28261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.450207 28257 heartbeater.cc:347] Connected to a master server at 127.25.83.62:39855
I20240501 19:56:52.450330 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:52.450341 28257 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:52.450498 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:56:52.450553 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:52.450584 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593412450583 us; error 0 us; skew 500 ppm
I20240501 19:56:52.450632 28257 heartbeater.cc:511] Master 127.25.83.62:39855 requested a full tablet report, sending...
I20240501 19:56:52.451325 28128 ts_manager.cc:194] Registered new tserver with Master: dcce477835bb469d90f6914314c22286 (127.25.83.1:45151)
I20240501 19:56:52.451557 25932 webserver.cc:460] Webserver started at http://127.25.83.2:44861/ using document root <none> and password file <none>
I20240501 19:56:52.451706 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.451779 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.451874 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.452307 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-1-root/instance:
uuid: "3909b4072f1840899e38b14273e68919"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-mfp3"
I20240501 19:56:52.452386 28128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58600
I20240501 19:56:52.455385 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:52.457062 28269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.457578 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:52.457726 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-1-root
uuid: "3909b4072f1840899e38b14273e68919"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-mfp3"
I20240501 19:56:52.457830 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.475633 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.475944 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.476199 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:52.476600 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:52.476635 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.476678 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:52.476711 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.480589 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.2:37061
I20240501 19:56:52.480669 28332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.2:37061 every 8 connection(s)
I20240501 19:56:52.481348 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:52.483664 28337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:52.484143 28336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.484686 25932 server_base.cc:1034] running on GCE node
I20240501 19:56:52.485299 28333 heartbeater.cc:347] Connected to a master server at 127.25.83.62:39855
W20240501 19:56:52.485335 28339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:52.485384 28333 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:52.485548 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 19:56:52.485569 28333 heartbeater.cc:511] Master 127.25.83.62:39855 requested a full tablet report, sending...
W20240501 19:56:52.485589 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:56:52.485770 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593412485768 us; error 0 us; skew 500 ppm
I20240501 19:56:52.486191 28128 ts_manager.cc:194] Registered new tserver with Master: 3909b4072f1840899e38b14273e68919 (127.25.83.2:37061)
I20240501 19:56:52.486677 25932 webserver.cc:460] Webserver started at http://127.25.83.3:37901/ using document root <none> and password file <none>
I20240501 19:56:52.486806 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:52.486814 28128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58608
I20240501 19:56:52.486893 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:52.486964 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:52.487438 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-2-root/instance:
uuid: "fc40c0c1473f4798b22e280ea9038a8b"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-mfp3"
I20240501 19:56:52.488826 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:52.489594 28344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.489821 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:52.489897 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-2-root
uuid: "fc40c0c1473f4798b22e280ea9038a8b"
format_stamp: "Formatted at 2024-05-01 19:56:52 on dist-test-slave-mfp3"
I20240501 19:56:52.489961 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1714593357002142-25932-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:52.501119 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:52.501386 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:52.501627 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:52.501992 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:52.502027 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.502064 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:52.502096 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:52.506201 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.3:44753
I20240501 19:56:52.506258 28407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.3:44753 every 8 connection(s)
I20240501 19:56:52.510457 28408 heartbeater.cc:347] Connected to a master server at 127.25.83.62:39855
I20240501 19:56:52.510525 28408 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:52.510685 28408 heartbeater.cc:511] Master 127.25.83.62:39855 requested a full tablet report, sending...
I20240501 19:56:52.511198 28128 ts_manager.cc:194] Registered new tserver with Master: fc40c0c1473f4798b22e280ea9038a8b (127.25.83.3:44753)
I20240501 19:56:52.511642 28128 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58620
I20240501 19:56:52.512097 25932 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005522797s
I20240501 19:56:53.453807 28257 heartbeater.cc:503] Master 127.25.83.62:39855 was elected leader, sending a full tablet report...
I20240501 19:56:53.487578 28333 heartbeater.cc:503] Master 127.25.83.62:39855 was elected leader, sending a full tablet report...
I20240501 19:56:53.512395 28408 heartbeater.cc:503] Master 127.25.83.62:39855 was elected leader, sending a full tablet report...
I20240501 19:56:54.381877 28182 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:56.382249 28182 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:56:58.382644 28182 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:00.382980 28182 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:02.383371 28182 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:04.383730 28182 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:04.589782 25932 tablet_server.cc:178] TabletServer@127.25.83.1:0 shutting down...
I20240501 19:57:04.593058 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:04.604895 25932 tablet_server.cc:194] TabletServer@127.25.83.1:0 shutdown complete.
I20240501 19:57:04.606657 25932 tablet_server.cc:178] TabletServer@127.25.83.2:0 shutting down...
I20240501 19:57:04.610086 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:04.621742 25932 tablet_server.cc:194] TabletServer@127.25.83.2:0 shutdown complete.
I20240501 19:57:04.623404 25932 tablet_server.cc:178] TabletServer@127.25.83.3:0 shutting down...
I20240501 19:57:04.626794 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:04.638506 25932 tablet_server.cc:194] TabletServer@127.25.83.3:0 shutdown complete.
I20240501 19:57:04.640002 25932 master.cc:537] Master@127.25.83.62:39855 shutting down...
I20240501 19:57:04.643064 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:04.643200 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:04.643260 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7ad7bdb64747416a9eb0060ee4d838fb: stopping tablet replica
I20240501 19:57:04.655072 25932 master.cc:552] Master@127.25.83.62:39855 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12317 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20240501 19:57:04.660310 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:39407
I20240501 19:57:04.660602 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:04.662897 28412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:04.663652 28413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:04.664165 28415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:04.664767 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:04.664932 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:04.664973 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:04.664994 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593424664993 us; error 0 us; skew 500 ppm
I20240501 19:57:04.665874 25932 webserver.cc:460] Webserver started at http://127.25.83.62:39211/ using document root <none> and password file <none>
I20240501 19:57:04.666023 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:04.666074 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:04.666211 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:04.666594 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "8cf9a1863df34fc693b625b8f9a57cbd"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.667999 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:04.668857 28420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.669090 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.669153 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "8cf9a1863df34fc693b625b8f9a57cbd"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.669221 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:04.679875 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:04.680222 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:04.684152 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:39407
I20240501 19:57:04.685559 28472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:39407 every 8 connection(s)
I20240501 19:57:04.685971 28473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:04.687657 28473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd: Bootstrap starting.
I20240501 19:57:04.688313 28473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:04.689193 28473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd: No bootstrap required, opened a new log
I20240501 19:57:04.689488 28473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf9a1863df34fc693b625b8f9a57cbd" member_type: VOTER }
I20240501 19:57:04.689563 28473 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:04.689591 28473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf9a1863df34fc693b625b8f9a57cbd, State: Initialized, Role: FOLLOWER
I20240501 19:57:04.689677 28473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf9a1863df34fc693b625b8f9a57cbd" member_type: VOTER }
I20240501 19:57:04.689728 28473 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:57:04.689749 28473 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:57:04.689779 28473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:04.690390 28473 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf9a1863df34fc693b625b8f9a57cbd" member_type: VOTER }
I20240501 19:57:04.690496 28473 leader_election.cc:302] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cf9a1863df34fc693b625b8f9a57cbd; no voters:
I20240501 19:57:04.690618 28473 leader_election.cc:288] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:57:04.690743 28476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:04.690922 28476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 1 LEADER]: Becoming Leader. State: Replica: 8cf9a1863df34fc693b625b8f9a57cbd, State: Running, Role: LEADER
I20240501 19:57:04.691017 28473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:04.691040 28476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf9a1863df34fc693b625b8f9a57cbd" member_type: VOTER }
I20240501 19:57:04.691422 28477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cf9a1863df34fc693b625b8f9a57cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf9a1863df34fc693b625b8f9a57cbd" member_type: VOTER } }
I20240501 19:57:04.691457 28478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf9a1863df34fc693b625b8f9a57cbd. Latest consensus state: current_term: 1 leader_uuid: "8cf9a1863df34fc693b625b8f9a57cbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cf9a1863df34fc693b625b8f9a57cbd" member_type: VOTER } }
I20240501 19:57:04.691557 28478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:04.691823 28482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:04.691998 28477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:04.692588 28482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:04.692792 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:57:04.694187 28482 catalog_manager.cc:1347] Generated new cluster ID: b3f00889e5c742ecb949cead639d5e40
I20240501 19:57:04.694240 28482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:04.702706 28482 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:57:04.703172 28482 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:04.711987 28482 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd: Generated new TSK 0
I20240501 19:57:04.712109 28482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:04.724977 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:04.727226 28494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:04.727787 28497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:04.728401 25932 server_base.cc:1034] running on GCE node
W20240501 19:57:04.729028 28495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:04.729202 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:04.729249 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:04.729270 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593424729270 us; error 0 us; skew 500 ppm
I20240501 19:57:04.730098 25932 webserver.cc:460] Webserver started at http://127.25.83.1:45313/ using document root <none> and password file <none>
I20240501 19:57:04.730265 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:04.730314 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:04.730387 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:04.730736 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-0-root/instance:
uuid: "c0e685777bb84569a83aa00090012b2b"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.732088 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:57:04.732962 28502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.733182 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:04.733243 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-0-root
uuid: "c0e685777bb84569a83aa00090012b2b"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.733307 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:04.752262 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:04.752625 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:04.752899 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:04.753338 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:04.753376 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.753427 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:04.753456 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.757544 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.1:41657
I20240501 19:57:04.757611 28565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.1:41657 every 8 connection(s)
I20240501 19:57:04.758370 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:04.760856 28571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:04.761497 28570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:04.762153 25932 server_base.cc:1034] running on GCE node
W20240501 19:57:04.762540 28573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:04.762812 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:04.762869 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:04.762892 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593424762892 us; error 0 us; skew 500 ppm
I20240501 19:57:04.763154 28566 heartbeater.cc:347] Connected to a master server at 127.25.83.62:39407
I20240501 19:57:04.763247 28566 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:04.763484 28566 heartbeater.cc:511] Master 127.25.83.62:39407 requested a full tablet report, sending...
I20240501 19:57:04.763909 25932 webserver.cc:460] Webserver started at http://127.25.83.2:46597/ using document root <none> and password file <none>
I20240501 19:57:04.764067 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:04.764118 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:04.764184 28437 ts_manager.cc:194] Registered new tserver with Master: c0e685777bb84569a83aa00090012b2b (127.25.83.1:41657)
I20240501 19:57:04.764194 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:04.764633 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-1-root/instance:
uuid: "83264934c6824d01832b9c18e48c2a08"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.764957 28437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49856
I20240501 19:57:04.766295 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:04.767199 28578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.767483 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.767550 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-1-root
uuid: "83264934c6824d01832b9c18e48c2a08"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.767619 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:04.784062 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:04.784379 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:04.784636 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:04.785040 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:04.785075 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.785125 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:04.785156 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.789197 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.2:35883
I20240501 19:57:04.789255 28641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.2:35883 every 8 connection(s)
I20240501 19:57:04.789955 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:04.792403 28645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:04.793103 28646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:04.793624 28648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:04.793721 28642 heartbeater.cc:347] Connected to a master server at 127.25.83.62:39407
I20240501 19:57:04.793792 28642 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:04.793956 28642 heartbeater.cc:511] Master 127.25.83.62:39407 requested a full tablet report, sending...
I20240501 19:57:04.794240 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:04.794431 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:04.794473 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:04.794488 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593424794489 us; error 0 us; skew 500 ppm
I20240501 19:57:04.794581 28437 ts_manager.cc:194] Registered new tserver with Master: 83264934c6824d01832b9c18e48c2a08 (127.25.83.2:35883)
I20240501 19:57:04.795311 28437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49872
I20240501 19:57:04.795336 25932 webserver.cc:460] Webserver started at http://127.25.83.3:41591/ using document root <none> and password file <none>
I20240501 19:57:04.795512 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:04.795559 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:04.795670 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:04.796036 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-2-root/instance:
uuid: "401b9c6376da4074b73c36c7551940c0"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.797364 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.798180 28653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.798401 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.798465 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-2-root
uuid: "401b9c6376da4074b73c36c7551940c0"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.798527 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:04.814685 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:04.814985 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:04.815249 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:04.815635 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:04.815672 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.815724 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:04.815761 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.819931 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.3:41945
I20240501 19:57:04.819996 28716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.3:41945 every 8 connection(s)
I20240501 19:57:04.820590 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:04.823108 28721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:04.823603 28720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:04.824334 28717 heartbeater.cc:347] Connected to a master server at 127.25.83.62:39407
I20240501 19:57:04.824424 28717 heartbeater.cc:464] Registering TS with master...
W20240501 19:57:04.824491 28723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:04.824628 28717 heartbeater.cc:511] Master 127.25.83.62:39407 requested a full tablet report, sending...
I20240501 19:57:04.824884 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:04.825063 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:04.825112 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:04.825145 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593424825143 us; error 0 us; skew 500 ppm
I20240501 19:57:04.825246 28437 ts_manager.cc:194] Registered new tserver with Master: 401b9c6376da4074b73c36c7551940c0 (127.25.83.3:41945)
I20240501 19:57:04.825798 28437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49876
I20240501 19:57:04.825978 25932 webserver.cc:460] Webserver started at http://127.25.83.4:37075/ using document root <none> and password file <none>
I20240501 19:57:04.826138 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:04.826193 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:04.826262 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:04.826583 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-3-root/instance:
uuid: "0ffab72499d04f4d82d82dd8006a2294"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.827893 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:04.828716 28728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.828953 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:04.829016 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-3-root
uuid: "0ffab72499d04f4d82d82dd8006a2294"
format_stamp: "Formatted at 2024-05-01 19:57:04 on dist-test-slave-mfp3"
I20240501 19:57:04.829078 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1714593357002142-25932-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:04.855029 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:04.855585 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:04.855983 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:04.856518 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:04.856560 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.856616 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:04.856650 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:04.861398 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.4:41613
I20240501 19:57:04.861472 28791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.4:41613 every 8 connection(s)
I20240501 19:57:04.866222 28792 heartbeater.cc:347] Connected to a master server at 127.25.83.62:39407
I20240501 19:57:04.866341 28792 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:04.866757 28792 heartbeater.cc:511] Master 127.25.83.62:39407 requested a full tablet report, sending...
I20240501 19:57:04.867564 28437 ts_manager.cc:194] Registered new tserver with Master: 0ffab72499d04f4d82d82dd8006a2294 (127.25.83.4:41613)
I20240501 19:57:04.868147 28437 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49888
I20240501 19:57:04.868608 25932 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.006788801s
I20240501 19:57:05.766198 28566 heartbeater.cc:503] Master 127.25.83.62:39407 was elected leader, sending a full tablet report...
I20240501 19:57:05.796244 28642 heartbeater.cc:503] Master 127.25.83.62:39407 was elected leader, sending a full tablet report...
I20240501 19:57:05.826669 28717 heartbeater.cc:503] Master 127.25.83.62:39407 was elected leader, sending a full tablet report...
I20240501 19:57:05.869024 28792 heartbeater.cc:503] Master 127.25.83.62:39407 was elected leader, sending a full tablet report...
I20240501 19:57:05.899178 25932 test_util.cc:268] Using random seed: -468336986
I20240501 19:57:05.906004 28437 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49900:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240501 19:57:05.915104 28606 tablet_service.cc:1467] Processing CreateTablet for tablet 111b29f4811f47fbac66b888b08790dc (DEFAULT_TABLE table=test-workload [id=0b4766f8154d41a5976fe34892764ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.915419 28606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111b29f4811f47fbac66b888b08790dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.916298 28530 tablet_service.cc:1467] Processing CreateTablet for tablet 111b29f4811f47fbac66b888b08790dc (DEFAULT_TABLE table=test-workload [id=0b4766f8154d41a5976fe34892764ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.916514 28530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111b29f4811f47fbac66b888b08790dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.917788 28813 tablet_bootstrap.cc:492] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08: Bootstrap starting.
I20240501 19:57:05.918484 28681 tablet_service.cc:1467] Processing CreateTablet for tablet 111b29f4811f47fbac66b888b08790dc (DEFAULT_TABLE table=test-workload [id=0b4766f8154d41a5976fe34892764ac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:05.918730 28681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 111b29f4811f47fbac66b888b08790dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:05.918776 28813 tablet_bootstrap.cc:654] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.919793 28813 tablet_bootstrap.cc:492] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08: No bootstrap required, opened a new log
I20240501 19:57:05.919893 28813 ts_tablet_manager.cc:1367] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:05.920307 28813 raft_consensus.cc:348] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.920423 28813 raft_consensus.cc:374] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.920467 28813 raft_consensus.cc:729] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83264934c6824d01832b9c18e48c2a08, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.920650 28813 consensus_queue.cc:260] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.920858 28816 tablet_bootstrap.cc:492] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0: Bootstrap starting.
I20240501 19:57:05.921021 28813 ts_tablet_manager.cc:1398] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:05.921921 28816 tablet_bootstrap.cc:654] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.922272 28814 tablet_bootstrap.cc:492] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b: Bootstrap starting.
I20240501 19:57:05.922999 28816 tablet_bootstrap.cc:492] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0: No bootstrap required, opened a new log
I20240501 19:57:05.923089 28816 ts_tablet_manager.cc:1367] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:05.923130 28814 tablet_bootstrap.cc:654] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:05.923513 28816 raft_consensus.cc:348] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.923616 28816 raft_consensus.cc:374] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.923648 28816 raft_consensus.cc:729] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 401b9c6376da4074b73c36c7551940c0, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.923836 28816 consensus_queue.cc:260] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.924171 28814 tablet_bootstrap.cc:492] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b: No bootstrap required, opened a new log
I20240501 19:57:05.924232 28816 ts_tablet_manager.cc:1398] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:05.924254 28814 ts_tablet_manager.cc:1367] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:05.924624 28814 raft_consensus.cc:348] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.924710 28814 raft_consensus.cc:374] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:05.924731 28814 raft_consensus.cc:729] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0e685777bb84569a83aa00090012b2b, State: Initialized, Role: FOLLOWER
I20240501 19:57:05.924991 28814 consensus_queue.cc:260] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.925328 28814 ts_tablet_manager.cc:1398] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:05.930819 28817 raft_consensus.cc:482] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:05.930898 28817 raft_consensus.cc:504] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.931296 28817 leader_election.cc:288] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 401b9c6376da4074b73c36c7551940c0 (127.25.83.3:41945), c0e685777bb84569a83aa00090012b2b (127.25.83.1:41657)
I20240501 19:57:05.934566 28691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111b29f4811f47fbac66b888b08790dc" candidate_uuid: "83264934c6824d01832b9c18e48c2a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401b9c6376da4074b73c36c7551940c0" is_pre_election: true
I20240501 19:57:05.934696 28691 raft_consensus.cc:2454] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83264934c6824d01832b9c18e48c2a08 in term 0.
I20240501 19:57:05.934847 28540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111b29f4811f47fbac66b888b08790dc" candidate_uuid: "83264934c6824d01832b9c18e48c2a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e685777bb84569a83aa00090012b2b" is_pre_election: true
I20240501 19:57:05.934960 28540 raft_consensus.cc:2454] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83264934c6824d01832b9c18e48c2a08 in term 0.
I20240501 19:57:05.934986 28580 leader_election.cc:302] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 401b9c6376da4074b73c36c7551940c0, 83264934c6824d01832b9c18e48c2a08; no voters:
I20240501 19:57:05.935150 28817 raft_consensus.cc:2781] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:05.935210 28817 raft_consensus.cc:482] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:05.935245 28817 raft_consensus.cc:3037] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.935981 28817 raft_consensus.cc:504] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.936264 28817 leader_election.cc:288] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [CANDIDATE]: Term 1 election: Requested vote from peers 401b9c6376da4074b73c36c7551940c0 (127.25.83.3:41945), c0e685777bb84569a83aa00090012b2b (127.25.83.1:41657)
I20240501 19:57:05.936539 28540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111b29f4811f47fbac66b888b08790dc" candidate_uuid: "83264934c6824d01832b9c18e48c2a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0e685777bb84569a83aa00090012b2b"
I20240501 19:57:05.936628 28540 raft_consensus.cc:3037] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.936549 28691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "111b29f4811f47fbac66b888b08790dc" candidate_uuid: "83264934c6824d01832b9c18e48c2a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "401b9c6376da4074b73c36c7551940c0"
I20240501 19:57:05.936725 28691 raft_consensus.cc:3037] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:05.937364 28540 raft_consensus.cc:2454] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83264934c6824d01832b9c18e48c2a08 in term 1.
I20240501 19:57:05.937506 28691 raft_consensus.cc:2454] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83264934c6824d01832b9c18e48c2a08 in term 1.
I20240501 19:57:05.937613 28582 leader_election.cc:302] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83264934c6824d01832b9c18e48c2a08, c0e685777bb84569a83aa00090012b2b; no voters:
I20240501 19:57:05.937768 28817 raft_consensus.cc:2781] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:05.937942 28817 raft_consensus.cc:686] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 1 LEADER]: Becoming Leader. State: Replica: 83264934c6824d01832b9c18e48c2a08, State: Running, Role: LEADER
I20240501 19:57:05.938056 28817 consensus_queue.cc:237] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } }
I20240501 19:57:05.939426 28437 catalog_manager.cc:5484] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83264934c6824d01832b9c18e48c2a08 (127.25.83.2). New cstate: current_term: 1 leader_uuid: "83264934c6824d01832b9c18e48c2a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83264934c6824d01832b9c18e48c2a08" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 35883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 } health_report { overall_health: UNKNOWN } } }
W20240501 19:57:06.009943 28567 tablet.cc:2369] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:57:06.321004 28817 consensus_queue.cc:1035] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0e685777bb84569a83aa00090012b2b" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 41657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.323912 28817 consensus_queue.cc:1035] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "401b9c6376da4074b73c36c7551940c0" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 41945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:06.692974 28491 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:06.693300 28491 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:06.693368 28491 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:08.693701 28491 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:08.694041 28491 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:08.694139 28491 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:10.695044 28491 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:10.695976 28491 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:10.696370 28491 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:12.696913 28491 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:12.697225 28491 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:12.697288 28491 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:14.697685 28491 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:14.697966 28491 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:14.698015 28491 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:16.698323 28491 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:16.698627 28491 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:16.698684 28491 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:16.992703 25932 tablet_server.cc:178] TabletServer@127.25.83.1:0 shutting down...
I20240501 19:57:16.996454 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:16.996623 25932 tablet_replica.cc:330] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b: stopping tablet replica
I20240501 19:57:16.996734 25932 raft_consensus.cc:2229] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:16.996843 25932 raft_consensus.cc:2258] T 111b29f4811f47fbac66b888b08790dc P c0e685777bb84569a83aa00090012b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:17.009284 25932 tablet_server.cc:194] TabletServer@127.25.83.1:0 shutdown complete.
I20240501 19:57:17.011185 25932 tablet_server.cc:178] TabletServer@127.25.83.2:0 shutting down...
I20240501 19:57:17.014392 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:17.014546 25932 tablet_replica.cc:330] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08: stopping tablet replica
I20240501 19:57:17.014640 25932 raft_consensus.cc:2229] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:17.014818 25932 raft_consensus.cc:2258] T 111b29f4811f47fbac66b888b08790dc P 83264934c6824d01832b9c18e48c2a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:17.027076 25932 tablet_server.cc:194] TabletServer@127.25.83.2:0 shutdown complete.
I20240501 19:57:17.028908 25932 tablet_server.cc:178] TabletServer@127.25.83.3:0 shutting down...
I20240501 19:57:17.032183 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:17.032380 25932 tablet_replica.cc:330] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0: stopping tablet replica
I20240501 19:57:17.032485 25932 raft_consensus.cc:2229] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:17.032590 25932 raft_consensus.cc:2258] T 111b29f4811f47fbac66b888b08790dc P 401b9c6376da4074b73c36c7551940c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:17.044919 25932 tablet_server.cc:194] TabletServer@127.25.83.3:0 shutdown complete.
I20240501 19:57:17.046911 25932 tablet_server.cc:178] TabletServer@127.25.83.4:0 shutting down...
I20240501 19:57:17.050926 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:17.063316 25932 tablet_server.cc:194] TabletServer@127.25.83.4:0 shutdown complete.
I20240501 19:57:17.065321 25932 master.cc:537] Master@127.25.83.62:39407 shutting down...
I20240501 19:57:17.068616 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:17.068800 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:17.068877 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8cf9a1863df34fc693b625b8f9a57cbd: stopping tablet replica
I20240501 19:57:17.081396 25932 master.cc:552] Master@127.25.83.62:39407 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12427 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20240501 19:57:17.087771 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:41677
I20240501 19:57:17.088068 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:57:17.090556 25932 server_base.cc:1034] running on GCE node
W20240501 19:57:17.091038 28834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:17.091557 28837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:17.092284 28835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:17.092482 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:17.092531 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:17.092554 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593437092553 us; error 0 us; skew 500 ppm
I20240501 19:57:17.093400 25932 webserver.cc:460] Webserver started at http://127.25.83.62:46841/ using document root <none> and password file <none>
I20240501 19:57:17.093535 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:17.093582 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:17.093658 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:17.094012 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "2462fe74272d4373af886cd533301d74"
format_stamp: "Formatted at 2024-05-01 19:57:17 on dist-test-slave-mfp3"
I20240501 19:57:17.095356 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:17.096170 28842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.096364 25932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20240501 19:57:17.096427 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "2462fe74272d4373af886cd533301d74"
format_stamp: "Formatted at 2024-05-01 19:57:17 on dist-test-slave-mfp3"
I20240501 19:57:17.096488 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:17.103948 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:17.104226 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:17.107825 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:41677
I20240501 19:57:17.109710 28894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:41677 every 8 connection(s)
I20240501 19:57:17.110162 28895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:17.111727 28895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74: Bootstrap starting.
I20240501 19:57:17.112377 28895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:17.113261 28895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74: No bootstrap required, opened a new log
I20240501 19:57:17.113580 28895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2462fe74272d4373af886cd533301d74" member_type: VOTER }
I20240501 19:57:17.113662 28895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:17.113696 28895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2462fe74272d4373af886cd533301d74, State: Initialized, Role: FOLLOWER
I20240501 19:57:17.113818 28895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2462fe74272d4373af886cd533301d74" member_type: VOTER }
I20240501 19:57:17.113880 28895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:57:17.113929 28895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:57:17.113978 28895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:17.114629 28895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2462fe74272d4373af886cd533301d74" member_type: VOTER }
I20240501 19:57:17.114742 28895 leader_election.cc:302] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2462fe74272d4373af886cd533301d74; no voters:
I20240501 19:57:17.114902 28895 leader_election.cc:288] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:57:17.114998 28898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:17.115185 28898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 1 LEADER]: Becoming Leader. State: Replica: 2462fe74272d4373af886cd533301d74, State: Running, Role: LEADER
I20240501 19:57:17.115288 28895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:17.115305 28898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2462fe74272d4373af886cd533301d74" member_type: VOTER }
I20240501 19:57:17.115749 28899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2462fe74272d4373af886cd533301d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2462fe74272d4373af886cd533301d74" member_type: VOTER } }
I20240501 19:57:17.115789 28900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2462fe74272d4373af886cd533301d74. Latest consensus state: current_term: 1 leader_uuid: "2462fe74272d4373af886cd533301d74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2462fe74272d4373af886cd533301d74" member_type: VOTER } }
I20240501 19:57:17.115931 28900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:17.116181 28905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:17.116387 28899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:17.116878 28905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:17.117094 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:57:17.118525 28905 catalog_manager.cc:1347] Generated new cluster ID: 23f4fc8f565843b08321ae92b92b32ba
I20240501 19:57:17.118578 28905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:17.125031 28905 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:57:17.125507 28905 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:17.131312 28905 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74: Generated new TSK 0
I20240501 19:57:17.131450 28905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:17.132969 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:17.135130 28917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:17.135782 28916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:17.136381 28919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:17.136920 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:17.137048 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:17.137084 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:17.137099 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593437137099 us; error 0 us; skew 500 ppm
I20240501 19:57:17.137883 25932 webserver.cc:460] Webserver started at http://127.25.83.1:44421/ using document root <none> and password file <none>
I20240501 19:57:17.138052 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:17.138092 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:17.138190 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:17.138556 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-0-root/instance:
uuid: "7319dc1ad82b416f9e949907865fbc11"
format_stamp: "Formatted at 2024-05-01 19:57:17 on dist-test-slave-mfp3"
I20240501 19:57:17.139901 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:57:17.140767 28924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.140990 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:17.141053 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-0-root
uuid: "7319dc1ad82b416f9e949907865fbc11"
format_stamp: "Formatted at 2024-05-01 19:57:17 on dist-test-slave-mfp3"
I20240501 19:57:17.141117 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:17.157685 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:17.158636 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:17.159055 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:17.159673 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:17.159711 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.159759 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:17.159795 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.165527 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.1:37175
I20240501 19:57:17.165571 28987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.1:37175 every 8 connection(s)
I20240501 19:57:17.166375 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:17.168860 28993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:17.169375 28995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:17.170037 28992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:17.170584 28988 heartbeater.cc:347] Connected to a master server at 127.25.83.62:41677
I20240501 19:57:17.170675 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:17.170708 28988 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:17.170822 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:17.170863 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:17.170895 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593437170894 us; error 0 us; skew 500 ppm
I20240501 19:57:17.170955 28988 heartbeater.cc:511] Master 127.25.83.62:41677 requested a full tablet report, sending...
I20240501 19:57:17.171612 28859 ts_manager.cc:194] Registered new tserver with Master: 7319dc1ad82b416f9e949907865fbc11 (127.25.83.1:37175)
I20240501 19:57:17.171851 25932 webserver.cc:460] Webserver started at http://127.25.83.2:34121/ using document root <none> and password file <none>
I20240501 19:57:17.172010 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:17.172066 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:17.172137 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:17.172523 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-1-root/instance:
uuid: "1f453c69d2f6452aa323eb64dc4378bf"
format_stamp: "Formatted at 2024-05-01 19:57:17 on dist-test-slave-mfp3"
I20240501 19:57:17.172600 28859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35664
I20240501 19:57:17.174012 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:57:17.174868 29000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.175091 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:17.175148 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-1-root
uuid: "1f453c69d2f6452aa323eb64dc4378bf"
format_stamp: "Formatted at 2024-05-01 19:57:17 on dist-test-slave-mfp3"
I20240501 19:57:17.175213 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:17.180858 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:17.181125 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:17.181355 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:17.181718 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:17.181752 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.181782 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:17.181815 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.185633 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.2:36881
I20240501 19:57:17.185695 29063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.2:36881 every 8 connection(s)
I20240501 19:57:17.186368 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:17.188486 29068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:17.189175 29067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:17.189909 29070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:17.190184 29064 heartbeater.cc:347] Connected to a master server at 127.25.83.62:41677
I20240501 19:57:17.190263 29064 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:17.190307 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:17.190448 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 19:57:17.190471 29064 heartbeater.cc:511] Master 127.25.83.62:41677 requested a full tablet report, sending...
W20240501 19:57:17.190474 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:17.190526 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593437190526 us; error 0 us; skew 500 ppm
I20240501 19:57:17.190976 28859 ts_manager.cc:194] Registered new tserver with Master: 1f453c69d2f6452aa323eb64dc4378bf (127.25.83.2:36881)
I20240501 19:57:17.191337 25932 webserver.cc:460] Webserver started at http://127.25.83.3:33483/ using document root <none> and password file <none>
I20240501 19:57:17.191466 28859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35668
I20240501 19:57:17.191474 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:17.191565 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:17.191637 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:17.191992 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-2-root/instance:
uuid: "100d5a57ee194b35aed83a2a08770e38"
format_stamp: "Formatted at 2024-05-01 19:57:17 on dist-test-slave-mfp3"
I20240501 19:57:17.193260 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:57:17.194079 29075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.194386 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:17.194461 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-2-root
uuid: "100d5a57ee194b35aed83a2a08770e38"
format_stamp: "Formatted at 2024-05-01 19:57:17 on dist-test-slave-mfp3"
I20240501 19:57:17.194530 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1714593357002142-25932-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:17.211243 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:17.211580 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:17.211858 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:17.212297 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:17.212338 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.212388 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:17.212425 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:17.217314 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.3:43921
I20240501 19:57:17.217376 29138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.3:43921 every 8 connection(s)
I20240501 19:57:17.221309 29139 heartbeater.cc:347] Connected to a master server at 127.25.83.62:41677
I20240501 19:57:17.221378 29139 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:17.221549 29139 heartbeater.cc:511] Master 127.25.83.62:41677 requested a full tablet report, sending...
I20240501 19:57:17.222060 28859 ts_manager.cc:194] Registered new tserver with Master: 100d5a57ee194b35aed83a2a08770e38 (127.25.83.3:43921)
I20240501 19:57:17.222244 25932 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004483189s
I20240501 19:57:17.222707 28859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35670
I20240501 19:57:18.173457 28988 heartbeater.cc:503] Master 127.25.83.62:41677 was elected leader, sending a full tablet report...
I20240501 19:57:18.192374 29064 heartbeater.cc:503] Master 127.25.83.62:41677 was elected leader, sending a full tablet report...
I20240501 19:57:18.223492 29139 heartbeater.cc:503] Master 127.25.83.62:41677 was elected leader, sending a full tablet report...
I20240501 19:57:18.256198 25932 test_util.cc:268] Using random seed: -455979964
I20240501 19:57:18.262742 28859 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44468:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:57:18.263193 28859 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:18.272817 29103 tablet_service.cc:1467] Processing CreateTablet for tablet 55ca1092b8964df4a58827c1d7c87198 (DEFAULT_TABLE table=test-workload [id=ff5bbb31323e4a94a26d4bf7b15b52bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:18.273008 28952 tablet_service.cc:1467] Processing CreateTablet for tablet 55ca1092b8964df4a58827c1d7c87198 (DEFAULT_TABLE table=test-workload [id=ff5bbb31323e4a94a26d4bf7b15b52bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:18.273165 29103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ca1092b8964df4a58827c1d7c87198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:18.273145 29028 tablet_service.cc:1467] Processing CreateTablet for tablet 55ca1092b8964df4a58827c1d7c87198 (DEFAULT_TABLE table=test-workload [id=ff5bbb31323e4a94a26d4bf7b15b52bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:57:18.273275 28952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ca1092b8964df4a58827c1d7c87198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:18.273344 29028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55ca1092b8964df4a58827c1d7c87198. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:18.275398 29160 tablet_bootstrap.cc:492] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38: Bootstrap starting.
I20240501 19:57:18.275905 29161 tablet_bootstrap.cc:492] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf: Bootstrap starting.
I20240501 19:57:18.276232 29160 tablet_bootstrap.cc:654] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:18.276645 29161 tablet_bootstrap.cc:654] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:18.277141 29160 tablet_bootstrap.cc:492] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38: No bootstrap required, opened a new log
I20240501 19:57:18.277284 29160 ts_tablet_manager.cc:1367] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:18.277653 29161 tablet_bootstrap.cc:492] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf: No bootstrap required, opened a new log
I20240501 19:57:18.277292 29162 tablet_bootstrap.cc:492] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11: Bootstrap starting.
I20240501 19:57:18.277714 29160 raft_consensus.cc:348] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.277832 29160 raft_consensus.cc:374] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:18.277876 29160 raft_consensus.cc:729] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 100d5a57ee194b35aed83a2a08770e38, State: Initialized, Role: FOLLOWER
I20240501 19:57:18.278012 29160 consensus_queue.cc:260] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.278385 29160 ts_tablet_manager.cc:1398] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:18.278638 29162 tablet_bootstrap.cc:654] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:18.277721 29161 ts_tablet_manager.cc:1367] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:18.279546 29161 raft_consensus.cc:348] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.279649 29161 raft_consensus.cc:374] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:18.279716 29161 raft_consensus.cc:729] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f453c69d2f6452aa323eb64dc4378bf, State: Initialized, Role: FOLLOWER
I20240501 19:57:18.279722 29162 tablet_bootstrap.cc:492] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11: No bootstrap required, opened a new log
I20240501 19:57:18.279911 29162 ts_tablet_manager.cc:1367] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:57:18.279907 29161 consensus_queue.cc:260] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.280279 29161 ts_tablet_manager.cc:1398] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:18.280386 29162 raft_consensus.cc:348] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.280475 29162 raft_consensus.cc:374] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:18.280730 29162 raft_consensus.cc:729] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7319dc1ad82b416f9e949907865fbc11, State: Initialized, Role: FOLLOWER
I20240501 19:57:18.280867 29162 consensus_queue.cc:260] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.281203 29162 ts_tablet_manager.cc:1398] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:18.306270 29167 raft_consensus.cc:482] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:18.306365 29167 raft_consensus.cc:504] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.306830 29167 leader_election.cc:288] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 100d5a57ee194b35aed83a2a08770e38 (127.25.83.3:43921), 7319dc1ad82b416f9e949907865fbc11 (127.25.83.1:37175)
I20240501 19:57:18.310154 29113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ca1092b8964df4a58827c1d7c87198" candidate_uuid: "1f453c69d2f6452aa323eb64dc4378bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100d5a57ee194b35aed83a2a08770e38" is_pre_election: true
I20240501 19:57:18.310206 28962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ca1092b8964df4a58827c1d7c87198" candidate_uuid: "1f453c69d2f6452aa323eb64dc4378bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7319dc1ad82b416f9e949907865fbc11" is_pre_election: true
I20240501 19:57:18.310264 29113 raft_consensus.cc:2454] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f453c69d2f6452aa323eb64dc4378bf in term 0.
I20240501 19:57:18.310336 28962 raft_consensus.cc:2454] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f453c69d2f6452aa323eb64dc4378bf in term 0.
I20240501 19:57:18.310528 29002 leader_election.cc:302] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100d5a57ee194b35aed83a2a08770e38, 1f453c69d2f6452aa323eb64dc4378bf; no voters:
I20240501 19:57:18.310695 29167 raft_consensus.cc:2781] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:18.310751 29167 raft_consensus.cc:482] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:18.310791 29167 raft_consensus.cc:3037] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:18.311511 29167 raft_consensus.cc:504] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.311784 29167 leader_election.cc:288] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [CANDIDATE]: Term 1 election: Requested vote from peers 100d5a57ee194b35aed83a2a08770e38 (127.25.83.3:43921), 7319dc1ad82b416f9e949907865fbc11 (127.25.83.1:37175)
I20240501 19:57:18.312034 29113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ca1092b8964df4a58827c1d7c87198" candidate_uuid: "1f453c69d2f6452aa323eb64dc4378bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "100d5a57ee194b35aed83a2a08770e38"
I20240501 19:57:18.312055 28962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55ca1092b8964df4a58827c1d7c87198" candidate_uuid: "1f453c69d2f6452aa323eb64dc4378bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7319dc1ad82b416f9e949907865fbc11"
I20240501 19:57:18.312116 29113 raft_consensus.cc:3037] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:18.312148 28962 raft_consensus.cc:3037] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:18.312810 29113 raft_consensus.cc:2454] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f453c69d2f6452aa323eb64dc4378bf in term 1.
I20240501 19:57:18.312954 28962 raft_consensus.cc:2454] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f453c69d2f6452aa323eb64dc4378bf in term 1.
I20240501 19:57:18.313015 29002 leader_election.cc:302] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 100d5a57ee194b35aed83a2a08770e38, 1f453c69d2f6452aa323eb64dc4378bf; no voters:
I20240501 19:57:18.313170 29167 raft_consensus.cc:2781] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:18.313329 29167 raft_consensus.cc:686] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 1 LEADER]: Becoming Leader. State: Replica: 1f453c69d2f6452aa323eb64dc4378bf, State: Running, Role: LEADER
I20240501 19:57:18.313443 29167 consensus_queue.cc:237] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } }
I20240501 19:57:18.314747 28858 catalog_manager.cc:5484] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f453c69d2f6452aa323eb64dc4378bf (127.25.83.2). New cstate: current_term: 1 leader_uuid: "1f453c69d2f6452aa323eb64dc4378bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f453c69d2f6452aa323eb64dc4378bf" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 36881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:18.774798 29167 consensus_queue.cc:1035] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7319dc1ad82b416f9e949907865fbc11" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 37175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:18.778236 29167 consensus_queue.cc:1035] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "100d5a57ee194b35aed83a2a08770e38" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:19.117261 28913 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:19.117395 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:19.117563 28913 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:19.117664 28913 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:20.119243 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:21.117976 28913 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:21.118320 28913 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:21.118383 28913 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:21.120548 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:22.122100 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:23.118753 28913 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:23.119068 28913 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:23.119122 28913 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:23.123466 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:24.124897 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:25.119475 28913 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:25.119877 28913 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:25.119947 28913 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:25.126294 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:26.127812 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:27.120266 28913 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:27.120615 28913 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:27.120688 28913 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:27.129164 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:28.130677 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:29.121223 28913 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:29.121577 28913 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:29.121635 28913 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:29.132064 28907 placement_policy_util.cc:407] tablet 55ca1092b8964df4a58827c1d7c87198: detected majority of replicas (2 of 3) at location L0
I20240501 19:57:29.379871 25932 tablet_server.cc:178] TabletServer@127.25.83.1:0 shutting down...
I20240501 19:57:29.383147 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:29.383323 25932 tablet_replica.cc:330] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11: stopping tablet replica
I20240501 19:57:29.383416 25932 raft_consensus.cc:2229] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:29.383519 25932 raft_consensus.cc:2258] T 55ca1092b8964df4a58827c1d7c87198 P 7319dc1ad82b416f9e949907865fbc11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 19:57:29.393362 29003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.83.1:37175: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240501 19:57:29.394083 29003 consensus_peers.cc:485] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf -> Peer 7319dc1ad82b416f9e949907865fbc11 (127.25.83.1:37175): Couldn't send request to peer 7319dc1ad82b416f9e949907865fbc11. Status: Network error: Client connection negotiation failed: client connection to 127.25.83.1:37175: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:29.395922 25932 tablet_server.cc:194] TabletServer@127.25.83.1:0 shutdown complete.
I20240501 19:57:29.397678 25932 tablet_server.cc:178] TabletServer@127.25.83.2:0 shutting down...
I20240501 19:57:29.401510 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:29.401690 25932 tablet_replica.cc:330] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf: stopping tablet replica
I20240501 19:57:29.401803 25932 raft_consensus.cc:2229] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:29.401978 25932 raft_consensus.cc:2258] T 55ca1092b8964df4a58827c1d7c87198 P 1f453c69d2f6452aa323eb64dc4378bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.414580 25932 tablet_server.cc:194] TabletServer@127.25.83.2:0 shutdown complete.
I20240501 19:57:29.416539 25932 tablet_server.cc:178] TabletServer@127.25.83.3:0 shutting down...
I20240501 19:57:29.420009 25932 ts_tablet_manager.cc:1470] Shutting down tablet manager...
I20240501 19:57:29.420184 25932 tablet_replica.cc:330] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38: stopping tablet replica
I20240501 19:57:29.420291 25932 raft_consensus.cc:2229] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:29.420405 25932 raft_consensus.cc:2258] T 55ca1092b8964df4a58827c1d7c87198 P 100d5a57ee194b35aed83a2a08770e38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.432642 25932 tablet_server.cc:194] TabletServer@127.25.83.3:0 shutdown complete.
I20240501 19:57:29.434466 25932 master.cc:537] Master@127.25.83.62:41677 shutting down...
I20240501 19:57:29.437517 25932 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 1 LEADER]: Raft consensus shutting down.
I20240501 19:57:29.437661 25932 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:29.437723 25932 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2462fe74272d4373af886cd533301d74: stopping tablet replica
I20240501 19:57:29.451541 25932 master.cc:552] Master@127.25.83.62:41677 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12370 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20240501 19:57:29.458134 25932 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.25.83.62:34915
I20240501 19:57:29.458459 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:29.461127 29186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:29.461834 29188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:29.462523 29185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:29.462981 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:29.463122 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:29.463156 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:29.463171 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593449463170 us; error 0 us; skew 500 ppm
I20240501 19:57:29.469280 25932 webserver.cc:460] Webserver started at http://127.25.83.62:41497/ using document root <none> and password file <none>
I20240501 19:57:29.469419 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:29.469460 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:29.469520 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:29.469878 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/master-0-root/instance:
uuid: "d8efb465ea1c4892a8a7cd704e7385a6"
format_stamp: "Formatted at 2024-05-01 19:57:29 on dist-test-slave-mfp3"
I20240501 19:57:29.471299 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20240501 19:57:29.472157 29193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.472362 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:29.472426 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/master-0-root
uuid: "d8efb465ea1c4892a8a7cd704e7385a6"
format_stamp: "Formatted at 2024-05-01 19:57:29 on dist-test-slave-mfp3"
I20240501 19:57:29.472484 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:29.483112 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:29.483438 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:29.487244 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.62:34915
I20240501 19:57:29.489037 29245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.62:34915 every 8 connection(s)
I20240501 19:57:29.489418 29246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:29.491147 29246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6: Bootstrap starting.
I20240501 19:57:29.491832 29246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:29.492762 29246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6: No bootstrap required, opened a new log
I20240501 19:57:29.493083 29246 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8efb465ea1c4892a8a7cd704e7385a6" member_type: VOTER }
I20240501 19:57:29.493163 29246 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:29.493196 29246 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8efb465ea1c4892a8a7cd704e7385a6, State: Initialized, Role: FOLLOWER
I20240501 19:57:29.493312 29246 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8efb465ea1c4892a8a7cd704e7385a6" member_type: VOTER }
I20240501 19:57:29.493373 29246 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:57:29.493414 29246 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:57:29.493464 29246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:29.494077 29246 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8efb465ea1c4892a8a7cd704e7385a6" member_type: VOTER }
I20240501 19:57:29.494220 29246 leader_election.cc:302] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8efb465ea1c4892a8a7cd704e7385a6; no voters:
I20240501 19:57:29.494374 29246 leader_election.cc:288] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:57:29.494462 29249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:29.494650 29249 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [term 1 LEADER]: Becoming Leader. State: Replica: d8efb465ea1c4892a8a7cd704e7385a6, State: Running, Role: LEADER
I20240501 19:57:29.494768 29246 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:57:29.494786 29249 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8efb465ea1c4892a8a7cd704e7385a6" member_type: VOTER }
I20240501 19:57:29.495213 29250 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8efb465ea1c4892a8a7cd704e7385a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8efb465ea1c4892a8a7cd704e7385a6" member_type: VOTER } }
I20240501 19:57:29.495239 29251 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8efb465ea1c4892a8a7cd704e7385a6. Latest consensus state: current_term: 1 leader_uuid: "d8efb465ea1c4892a8a7cd704e7385a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8efb465ea1c4892a8a7cd704e7385a6" member_type: VOTER } }
I20240501 19:57:29.495385 29251 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:29.495640 29250 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6 [sys.catalog]: This master's current role is: LEADER
I20240501 19:57:29.495860 29256 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:57:29.496641 29256 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:57:29.496793 25932 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20240501 19:57:29.498256 29256 catalog_manager.cc:1347] Generated new cluster ID: e762e03248544d2690ea110fc14971b6
I20240501 19:57:29.498312 29256 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:57:29.511648 29256 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:57:29.512104 29256 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:57:29.517822 29256 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d8efb465ea1c4892a8a7cd704e7385a6: Generated new TSK 0
I20240501 19:57:29.517957 29256 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:57:29.528877 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:29.531191 29268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:29.531674 29270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:29.532178 29267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:29.532954 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:29.533106 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:29.533145 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:29.533161 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593449533161 us; error 0 us; skew 500 ppm
I20240501 19:57:29.534080 25932 webserver.cc:460] Webserver started at http://127.25.83.1:38753/ using document root <none> and password file <none>
I20240501 19:57:29.534247 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:29.534297 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:29.534371 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:29.534740 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root/instance:
uuid: "649b8f2f20324033bc021e229cff5774"
format_stamp: "Formatted at 2024-05-01 19:57:29 on dist-test-slave-mfp3"
I20240501 19:57:29.536068 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:29.536854 29275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.537091 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.537147 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root
uuid: "649b8f2f20324033bc021e229cff5774"
format_stamp: "Formatted at 2024-05-01 19:57:29 on dist-test-slave-mfp3"
I20240501 19:57:29.537194 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:29.544792 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:29.545047 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:29.545248 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:29.545598 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:29.545629 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.545656 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:29.545675 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.549305 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.1:46115
I20240501 19:57:29.549329 29338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.1:46115 every 8 connection(s)
I20240501 19:57:29.550030 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:29.552250 29343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:29.552799 29346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:29.553445 29344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:29.554193 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:29.554287 29339 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34915
I20240501 19:57:29.554368 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20240501 19:57:29.554371 29339 heartbeater.cc:464] Registering TS with master...
W20240501 19:57:29.554425 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:29.554445 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593449554444 us; error 0 us; skew 500 ppm
I20240501 19:57:29.554598 29339 heartbeater.cc:511] Master 127.25.83.62:34915 requested a full tablet report, sending...
I20240501 19:57:29.555215 29210 ts_manager.cc:194] Registered new tserver with Master: 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115)
I20240501 19:57:29.555353 25932 webserver.cc:460] Webserver started at http://127.25.83.2:40407/ using document root <none> and password file <none>
I20240501 19:57:29.555505 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:29.555557 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:29.555625 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:29.555997 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root/instance:
uuid: "26f1ddf6a31645c0a82e697acc9253fb"
format_stamp: "Formatted at 2024-05-01 19:57:29 on dist-test-slave-mfp3"
I20240501 19:57:29.556015 29210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56528
I20240501 19:57:29.557381 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:29.558208 29351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.558400 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.558465 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root
uuid: "26f1ddf6a31645c0a82e697acc9253fb"
format_stamp: "Formatted at 2024-05-01 19:57:29 on dist-test-slave-mfp3"
I20240501 19:57:29.558529 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:29.571763 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:29.572158 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:29.572482 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:29.572942 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:29.572983 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.573014 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:29.573035 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.577497 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.2:32945
I20240501 19:57:29.577530 29414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.2:32945 every 8 connection(s)
I20240501 19:57:29.579062 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:29.582957 29419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:29.585479 29415 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34915
I20240501 19:57:29.585729 29415 heartbeater.cc:464] Registering TS with master...
W20240501 19:57:29.585821 29421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:29.586163 29415 heartbeater.cc:511] Master 127.25.83.62:34915 requested a full tablet report, sending...
W20240501 19:57:29.585547 29418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:29.585754 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:29.586750 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:29.586820 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:29.586844 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593449586844 us; error 0 us; skew 500 ppm
I20240501 19:57:29.586972 29210 ts_manager.cc:194] Registered new tserver with Master: 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:29.587693 29210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56532
I20240501 19:57:29.587931 25932 webserver.cc:460] Webserver started at http://127.25.83.3:45647/ using document root <none> and password file <none>
I20240501 19:57:29.588089 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:29.588155 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:29.588230 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:29.588613 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root/instance:
uuid: "77ff839c08d0479993ba43e53980437b"
format_stamp: "Formatted at 2024-05-01 19:57:29 on dist-test-slave-mfp3"
I20240501 19:57:29.589937 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:29.590806 29426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.591022 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:29.591075 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root
uuid: "77ff839c08d0479993ba43e53980437b"
format_stamp: "Formatted at 2024-05-01 19:57:29 on dist-test-slave-mfp3"
I20240501 19:57:29.591131 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:29.604255 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:29.604588 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:29.604837 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:29.605209 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:29.605245 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.605288 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:29.605316 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:29.609597 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.3:44839
I20240501 19:57:29.609649 29489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.3:44839 every 8 connection(s)
I20240501 19:57:29.613457 29490 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34915
I20240501 19:57:29.613533 29490 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:29.613723 29490 heartbeater.cc:511] Master 127.25.83.62:34915 requested a full tablet report, sending...
I20240501 19:57:29.614254 29210 ts_manager.cc:194] Registered new tserver with Master: 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:29.614380 25932 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.004391869s
I20240501 19:57:29.614696 29210 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56536
I20240501 19:57:30.556983 29339 heartbeater.cc:503] Master 127.25.83.62:34915 was elected leader, sending a full tablet report...
I20240501 19:57:30.588675 29415 heartbeater.cc:503] Master 127.25.83.62:34915 was elected leader, sending a full tablet report...
I20240501 19:57:30.615525 29490 heartbeater.cc:503] Master 127.25.83.62:34915 was elected leader, sending a full tablet report...
I20240501 19:57:30.640147 25932 test_util.cc:268] Using random seed: -443596016
I20240501 19:57:30.647944 29210 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56550:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240501 19:57:30.648543 29210 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240501 19:57:30.661690 29453 tablet_service.cc:1467] Processing CreateTablet for tablet b886671d00534706ad4fc5f6023c1dd0 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 19:57:30.662081 29453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b886671d00534706ad4fc5f6023c1dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.662359 29379 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe56fe5bfe2443c97aa252754a24aa6 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 19:57:30.662513 29377 tablet_service.cc:1467] Processing CreateTablet for tablet 2803395236b144059af4f49d49e8c7e3 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 19:57:30.662631 29379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe56fe5bfe2443c97aa252754a24aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.662752 29377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2803395236b144059af4f49d49e8c7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.663007 29378 tablet_service.cc:1467] Processing CreateTablet for tablet b886671d00534706ad4fc5f6023c1dd0 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 19:57:30.663218 29378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b886671d00534706ad4fc5f6023c1dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.664850 29452 tablet_service.cc:1467] Processing CreateTablet for tablet 2803395236b144059af4f49d49e8c7e3 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 19:57:30.665119 29452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2803395236b144059af4f49d49e8c7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.665324 29376 tablet_service.cc:1467] Processing CreateTablet for tablet 9967547056d848d1b37aff3f6b4abe86 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 19:57:30.665518 29376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9967547056d848d1b37aff3f6b4abe86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.666757 29451 tablet_service.cc:1467] Processing CreateTablet for tablet 9967547056d848d1b37aff3f6b4abe86 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 19:57:30.666965 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9967547056d848d1b37aff3f6b4abe86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.667057 29375 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd1acf0afe14cc2a388fcf79ddae20d (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 19:57:30.667235 29375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd1acf0afe14cc2a388fcf79ddae20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.668541 29450 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd1acf0afe14cc2a388fcf79ddae20d (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 19:57:30.668810 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd1acf0afe14cc2a388fcf79ddae20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.668778 29374 tablet_service.cc:1467] Processing CreateTablet for tablet 3e50e51f9e3c4d3d83ae49f7a326551a (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 19:57:30.669057 29374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e50e51f9e3c4d3d83ae49f7a326551a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.669025 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 3e50e51f9e3c4d3d83ae49f7a326551a (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 19:57:30.668860 29373 tablet_service.cc:1467] Processing CreateTablet for tablet a2f89b5db8c34edaa8751a263dc818cd (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 19:57:30.669270 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e50e51f9e3c4d3d83ae49f7a326551a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.669319 29373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f89b5db8c34edaa8751a263dc818cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.670753 29511 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.661690 29454 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe56fe5bfe2443c97aa252754a24aa6 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 19:57:30.671804 29511 tablet_bootstrap.cc:654] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.671967 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe56fe5bfe2443c97aa252754a24aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.673523 29511 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.673609 29511 ts_tablet_manager.cc:1367] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:30.674082 29511 raft_consensus.cc:348] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.674310 29511 raft_consensus.cc:374] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.674398 29511 raft_consensus.cc:729] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.674599 29511 consensus_queue.cc:260] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.674679 29303 tablet_service.cc:1467] Processing CreateTablet for tablet 5fe56fe5bfe2443c97aa252754a24aa6 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION VALUES < 178956970
I20240501 19:57:30.674636 29454 tablet_service.cc:1467] Processing CreateTablet for tablet a2f89b5db8c34edaa8751a263dc818cd (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 19:57:30.674908 29303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe56fe5bfe2443c97aa252754a24aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.674908 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f89b5db8c34edaa8751a263dc818cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.674973 29299 tablet_service.cc:1467] Processing CreateTablet for tablet 8cd1acf0afe14cc2a388fcf79ddae20d (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20240501 19:57:30.675180 29511 ts_tablet_manager.cc:1398] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.675402 29300 tablet_service.cc:1467] Processing CreateTablet for tablet 9967547056d848d1b37aff3f6b4abe86 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20240501 19:57:30.675671 29511 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.675812 29301 tablet_service.cc:1467] Processing CreateTablet for tablet 2803395236b144059af4f49d49e8c7e3 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20240501 19:57:30.676223 29302 tablet_service.cc:1467] Processing CreateTablet for tablet b886671d00534706ad4fc5f6023c1dd0 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20240501 19:57:30.675190 29299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cd1acf0afe14cc2a388fcf79ddae20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.676575 29511 tablet_bootstrap.cc:654] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.677155 29514 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.677260 29298 tablet_service.cc:1467] Processing CreateTablet for tablet 3e50e51f9e3c4d3d83ae49f7a326551a (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20240501 19:57:30.677495 29298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e50e51f9e3c4d3d83ae49f7a326551a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.677613 29511 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.677683 29511 ts_tablet_manager.cc:1367] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:30.678103 29514 tablet_bootstrap.cc:654] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.678084 29511 raft_consensus.cc:348] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.678217 29511 raft_consensus.cc:374] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.678251 29511 raft_consensus.cc:729] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.678364 29511 consensus_queue.cc:260] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.678429 29302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b886671d00534706ad4fc5f6023c1dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.678726 29511 ts_tablet_manager.cc:1398] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.678889 29511 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.679703 29511 tablet_bootstrap.cc:654] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.679653 29297 tablet_service.cc:1467] Processing CreateTablet for tablet a2f89b5db8c34edaa8751a263dc818cd (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20240501 19:57:30.679836 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9967547056d848d1b37aff3f6b4abe86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.680027 29296 tablet_service.cc:1467] Processing CreateTablet for tablet ffed8312f8034d9fa6aa423ced8117ee (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 19:57:30.680262 29296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffed8312f8034d9fa6aa423ced8117ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.680486 29297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f89b5db8c34edaa8751a263dc818cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.680730 29511 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.680804 29511 ts_tablet_manager.cc:1367] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.681205 29511 raft_consensus.cc:348] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.681294 29511 raft_consensus.cc:374] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.681324 29511 raft_consensus.cc:729] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.681432 29511 consensus_queue.cc:260] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.681704 29511 ts_tablet_manager.cc:1398] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.681746 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2803395236b144059af4f49d49e8c7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.681849 29511 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.682739 29511 tablet_bootstrap.cc:654] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.684046 29510 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.684607 29511 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.684684 29511 ts_tablet_manager.cc:1367] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:30.684990 29510 tablet_bootstrap.cc:654] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.685091 29511 raft_consensus.cc:348] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.685207 29511 raft_consensus.cc:374] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.685256 29511 raft_consensus.cc:729] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.685372 29511 consensus_queue.cc:260] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.685424 29374 tablet_service.cc:1467] Processing CreateTablet for tablet ffed8312f8034d9fa6aa423ced8117ee (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 19:57:30.685617 29511 ts_tablet_manager.cc:1398] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.685446 29454 tablet_service.cc:1467] Processing CreateTablet for tablet ffed8312f8034d9fa6aa423ced8117ee (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20240501 19:57:30.685691 29374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffed8312f8034d9fa6aa423ced8117ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.685762 29511 tablet_bootstrap.cc:492] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.685842 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffed8312f8034d9fa6aa423ced8117ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.686196 29514 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.686268 29514 ts_tablet_manager.cc:1367] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:57:30.686673 29511 tablet_bootstrap.cc:654] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.686648 29514 raft_consensus.cc:348] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.686731 29514 raft_consensus.cc:374] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.686762 29514 raft_consensus.cc:729] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.686903 29514 consensus_queue.cc:260] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.687204 29514 ts_tablet_manager.cc:1398] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.688194 29514 tablet_bootstrap.cc:492] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.688508 29296 tablet_service.cc:1467] Processing CreateTablet for tablet c0d7f921a55e437eaaddf7d276c8de0c (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 19:57:30.688750 29296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d7f921a55e437eaaddf7d276c8de0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.689103 29514 tablet_bootstrap.cc:654] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.689294 29510 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.689383 29510 ts_tablet_manager.cc:1367] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:57:30.689780 29510 raft_consensus.cc:348] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.689924 29510 raft_consensus.cc:374] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.690280 29374 tablet_service.cc:1467] Processing CreateTablet for tablet c0d7f921a55e437eaaddf7d276c8de0c (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 19:57:30.690305 29454 tablet_service.cc:1467] Processing CreateTablet for tablet c0d7f921a55e437eaaddf7d276c8de0c (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20240501 19:57:30.690366 29297 tablet_service.cc:1467] Processing CreateTablet for tablet 39fb3d9c368d4c0d95bf935f60f863e8 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 19:57:30.690495 29374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d7f921a55e437eaaddf7d276c8de0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.690515 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d7f921a55e437eaaddf7d276c8de0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.690945 29301 tablet_service.cc:1467] Processing CreateTablet for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 19:57:30.690528 29297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39fb3d9c368d4c0d95bf935f60f863e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.691546 29300 tablet_service.cc:1467] Processing CreateTablet for tablet e0518736faac4d54bdb78737484c860a (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 19:57:30.691736 29300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0518736faac4d54bdb78737484c860a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.692713 29449 tablet_service.cc:1467] Processing CreateTablet for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 19:57:30.692876 29301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.692899 29373 tablet_service.cc:1467] Processing CreateTablet for tablet 39fb3d9c368d4c0d95bf935f60f863e8 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 19:57:30.692960 29449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.693104 29373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39fb3d9c368d4c0d95bf935f60f863e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.694356 29450 tablet_service.cc:1467] Processing CreateTablet for tablet e0518736faac4d54bdb78737484c860a (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 19:57:30.694571 29450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0518736faac4d54bdb78737484c860a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.695109 29375 tablet_service.cc:1467] Processing CreateTablet for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20240501 19:57:30.695345 29375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.695415 29454 tablet_service.cc:1467] Processing CreateTablet for tablet 39fb3d9c368d4c0d95bf935f60f863e8 (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20240501 19:57:30.695678 29454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39fb3d9c368d4c0d95bf935f60f863e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.689986 29510 raft_consensus.cc:729] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.696175 29510 consensus_queue.cc:260] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.696746 29376 tablet_service.cc:1467] Processing CreateTablet for tablet e0518736faac4d54bdb78737484c860a (DEFAULT_TABLE table=test-workload [id=9750aeea18d542b4b06c0492a5d30987]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20240501 19:57:30.696796 29510 ts_tablet_manager.cc:1398] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:57:30.696981 29376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0518736faac4d54bdb78737484c860a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:30.697490 29510 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.698603 29511 tablet_bootstrap.cc:492] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.698704 29511 ts_tablet_manager.cc:1367] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:57:30.699112 29511 raft_consensus.cc:348] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.699225 29511 raft_consensus.cc:374] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.699260 29511 raft_consensus.cc:729] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.699381 29511 consensus_queue.cc:260] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.699591 29514 tablet_bootstrap.cc:492] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.699627 29511 ts_tablet_manager.cc:1398] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.699664 29514 ts_tablet_manager.cc:1367] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240501 19:57:30.699728 29510 tablet_bootstrap.cc:654] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.699796 29511 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.700062 29514 raft_consensus.cc:348] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.700170 29514 raft_consensus.cc:374] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.700204 29514 raft_consensus.cc:729] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.700340 29514 consensus_queue.cc:260] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.700583 29514 ts_tablet_manager.cc:1398] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.700651 29511 tablet_bootstrap.cc:654] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.700793 29510 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.700875 29510 ts_tablet_manager.cc:1367] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:30.701061 29514 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.701279 29510 raft_consensus.cc:348] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.701383 29510 raft_consensus.cc:374] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.701417 29510 raft_consensus.cc:729] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.701522 29510 consensus_queue.cc:260] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.701664 29511 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.701731 29511 ts_tablet_manager.cc:1367] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.701742 29510 ts_tablet_manager.cc:1398] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.701895 29510 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.702103 29511 raft_consensus.cc:348] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.702222 29511 raft_consensus.cc:374] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.702255 29511 raft_consensus.cc:729] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.702551 29511 consensus_queue.cc:260] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.701900 29514 tablet_bootstrap.cc:654] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.702895 29511 ts_tablet_manager.cc:1398] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.703104 29511 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.702709 29510 tablet_bootstrap.cc:654] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.704028 29511 tablet_bootstrap.cc:654] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.704864 29514 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.704948 29514 ts_tablet_manager.cc:1367] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:30.705327 29510 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.705406 29510 ts_tablet_manager.cc:1367] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:30.705346 29514 raft_consensus.cc:348] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.705662 29514 raft_consensus.cc:374] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.705713 29514 raft_consensus.cc:729] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.705792 29510 raft_consensus.cc:348] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.705883 29510 raft_consensus.cc:374] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.705927 29510 raft_consensus.cc:729] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.705822 29514 consensus_queue.cc:260] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.706034 29510 consensus_queue.cc:260] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.706254 29514 ts_tablet_manager.cc:1398] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.706315 29511 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.706331 29510 ts_tablet_manager.cc:1398] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.706374 29511 ts_tablet_manager.cc:1367] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:30.706395 29514 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.706476 29510 tablet_bootstrap.cc:492] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.706748 29511 raft_consensus.cc:348] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.706840 29511 raft_consensus.cc:374] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.706874 29511 raft_consensus.cc:729] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.706982 29511 consensus_queue.cc:260] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.707180 29510 tablet_bootstrap.cc:654] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.707238 29514 tablet_bootstrap.cc:654] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.707264 29511 ts_tablet_manager.cc:1398] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:30.707412 29511 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.708271 29511 tablet_bootstrap.cc:654] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.709194 29510 tablet_bootstrap.cc:492] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.709278 29510 ts_tablet_manager.cc:1367] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:30.709357 29511 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.709362 29514 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.709442 29511 ts_tablet_manager.cc:1367] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.709501 29514 ts_tablet_manager.cc:1367] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:30.709686 29510 raft_consensus.cc:348] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.709848 29510 raft_consensus.cc:374] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.709887 29511 raft_consensus.cc:348] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.709939 29514 raft_consensus.cc:348] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.710039 29511 raft_consensus.cc:374] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.710086 29514 raft_consensus.cc:374] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.710144 29511 raft_consensus.cc:729] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.709942 29510 raft_consensus.cc:729] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.710350 29511 consensus_queue.cc:260] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.710387 29510 consensus_queue.cc:260] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.710670 29510 ts_tablet_manager.cc:1398] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.710742 29511 ts_tablet_manager.cc:1398] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.710813 29510 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.710887 29511 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.711583 29510 tablet_bootstrap.cc:654] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.711752 29511 tablet_bootstrap.cc:654] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.710180 29514 raft_consensus.cc:729] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.712080 29514 consensus_queue.cc:260] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.712352 29514 ts_tablet_manager.cc:1398] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:30.712509 29514 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.713398 29514 tablet_bootstrap.cc:654] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.714810 29510 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.714910 29510 ts_tablet_manager.cc:1367] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:30.714987 29511 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.715077 29511 ts_tablet_manager.cc:1367] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:30.715265 29510 raft_consensus.cc:348] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.715338 29510 raft_consensus.cc:374] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.715359 29510 raft_consensus.cc:729] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.715396 29511 raft_consensus.cc:348] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.715484 29511 raft_consensus.cc:374] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.715469 29510 consensus_queue.cc:260] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.715525 29511 raft_consensus.cc:729] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.715723 29510 ts_tablet_manager.cc:1398] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.715863 29510 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.716104 29511 consensus_queue.cc:260] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.716447 29511 ts_tablet_manager.cc:1398] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.716643 29511 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.716710 29510 tablet_bootstrap.cc:654] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.717584 29511 tablet_bootstrap.cc:654] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.717794 29510 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.717875 29510 ts_tablet_manager.cc:1367] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.718312 29514 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.718394 29514 ts_tablet_manager.cc:1367] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:57:30.718375 29510 raft_consensus.cc:348] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.718519 29510 raft_consensus.cc:374] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.718562 29510 raft_consensus.cc:729] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.718678 29510 consensus_queue.cc:260] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.718794 29514 raft_consensus.cc:348] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.718899 29514 raft_consensus.cc:374] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.718945 29514 raft_consensus.cc:729] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.718946 29510 ts_tablet_manager.cc:1398] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.719126 29510 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.719329 29514 consensus_queue.cc:260] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.719610 29514 ts_tablet_manager.cc:1398] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.719950 29510 tablet_bootstrap.cc:654] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.719964 29511 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.720090 29511 ts_tablet_manager.cc:1367] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:30.720274 29514 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.720974 29510 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.721050 29510 ts_tablet_manager.cc:1367] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:30.721307 29514 tablet_bootstrap.cc:654] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.721338 29511 raft_consensus.cc:348] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.721407 29511 raft_consensus.cc:374] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.721402 29510 raft_consensus.cc:348] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.721478 29511 raft_consensus.cc:729] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.721503 29510 raft_consensus.cc:374] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.721534 29510 raft_consensus.cc:729] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.721626 29511 consensus_queue.cc:260] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.722297 29514 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.722376 29514 ts_tablet_manager.cc:1367] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:30.722470 29511 ts_tablet_manager.cc:1398] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.722599 29511 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.722682 29510 consensus_queue.cc:260] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.722725 29514 raft_consensus.cc:348] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.722798 29514 raft_consensus.cc:374] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.722826 29514 raft_consensus.cc:729] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.722937 29514 consensus_queue.cc:260] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.723205 29514 ts_tablet_manager.cc:1398] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.723356 29514 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.723675 29511 tablet_bootstrap.cc:654] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.724211 29514 tablet_bootstrap.cc:654] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.724408 29510 ts_tablet_manager.cc:1398] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:30.724587 29510 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.725512 29514 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.725589 29514 ts_tablet_manager.cc:1367] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.725889 29514 raft_consensus.cc:348] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.725962 29514 raft_consensus.cc:374] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.725973 29510 tablet_bootstrap.cc:654] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.725996 29514 raft_consensus.cc:729] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.726203 29514 consensus_queue.cc:260] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.726485 29514 ts_tablet_manager.cc:1398] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:30.726665 29514 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.726752 29511 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.726825 29511 ts_tablet_manager.cc:1367] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:30.727042 29510 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.727123 29510 ts_tablet_manager.cc:1367] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:30.727221 29511 raft_consensus.cc:348] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.727332 29511 raft_consensus.cc:374] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.727409 29511 raft_consensus.cc:729] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.727488 29510 raft_consensus.cc:348] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.727515 29514 tablet_bootstrap.cc:654] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.727566 29511 consensus_queue.cc:260] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.728179 29511 ts_tablet_manager.cc:1398] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.728387 29511 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: Bootstrap starting.
I20240501 19:57:30.728720 29514 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.728829 29514 ts_tablet_manager.cc:1367] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:57:30.727577 29510 raft_consensus.cc:374] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.729171 29510 raft_consensus.cc:729] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.729305 29511 tablet_bootstrap.cc:654] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.729243 29514 raft_consensus.cc:348] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.729369 29510 consensus_queue.cc:260] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.729419 29514 raft_consensus.cc:374] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.729458 29514 raft_consensus.cc:729] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.729593 29514 consensus_queue.cc:260] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.729876 29514 ts_tablet_manager.cc:1398] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:30.730033 29514 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.730643 29511 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: No bootstrap required, opened a new log
I20240501 19:57:30.730707 29511 ts_tablet_manager.cc:1367] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.730881 29514 tablet_bootstrap.cc:654] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.730971 29510 ts_tablet_manager.cc:1398] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:57:30.731140 29510 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.731115 29511 raft_consensus.cc:348] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.731211 29511 raft_consensus.cc:374] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.731245 29511 raft_consensus.cc:729] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.731390 29511 consensus_queue.cc:260] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.732038 29510 tablet_bootstrap.cc:654] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.732573 29511 ts_tablet_manager.cc:1398] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.733072 29510 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.733175 29510 ts_tablet_manager.cc:1367] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:57:30.733541 29510 raft_consensus.cc:348] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.733635 29510 raft_consensus.cc:374] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.733666 29510 raft_consensus.cc:729] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.733784 29510 consensus_queue.cc:260] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.734030 29510 ts_tablet_manager.cc:1398] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.734179 29514 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.734210 29510 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.734256 29514 ts_tablet_manager.cc:1367] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:57:30.734565 29514 raft_consensus.cc:348] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.734647 29514 raft_consensus.cc:374] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.734688 29514 raft_consensus.cc:729] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.734802 29514 consensus_queue.cc:260] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.735013 29514 ts_tablet_manager.cc:1398] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.735060 29510 tablet_bootstrap.cc:654] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.735163 29514 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.736084 29514 tablet_bootstrap.cc:654] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.736485 29510 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.736568 29510 ts_tablet_manager.cc:1367] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.736958 29510 raft_consensus.cc:348] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.737066 29510 raft_consensus.cc:374] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.737108 29510 raft_consensus.cc:729] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.737216 29510 consensus_queue.cc:260] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.737483 29510 ts_tablet_manager.cc:1398] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.737634 29510 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.738027 29514 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.738121 29514 ts_tablet_manager.cc:1367] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:57:30.738473 29514 raft_consensus.cc:348] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.738551 29514 raft_consensus.cc:374] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.738586 29510 tablet_bootstrap.cc:654] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.738586 29514 raft_consensus.cc:729] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.738759 29514 consensus_queue.cc:260] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.739003 29514 ts_tablet_manager.cc:1398] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:30.739233 29514 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.739616 29510 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.739709 29510 ts_tablet_manager.cc:1367] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:57:30.739931 29514 tablet_bootstrap.cc:654] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.740069 29510 raft_consensus.cc:348] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.740168 29510 raft_consensus.cc:374] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.740211 29510 raft_consensus.cc:729] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.740322 29510 consensus_queue.cc:260] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.740581 29510 ts_tablet_manager.cc:1398] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:30.740760 29510 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b: Bootstrap starting.
I20240501 19:57:30.741312 29514 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.741389 29514 ts_tablet_manager.cc:1367] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:57:30.741685 29510 tablet_bootstrap.cc:654] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.741755 29514 raft_consensus.cc:348] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.741818 29514 raft_consensus.cc:374] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.741837 29514 raft_consensus.cc:729] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.741919 29514 consensus_queue.cc:260] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.742373 29514 ts_tablet_manager.cc:1398] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:30.742530 29514 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: Bootstrap starting.
I20240501 19:57:30.744302 29514 tablet_bootstrap.cc:654] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: Neither blocks nor log segments found. Creating new log.
I20240501 19:57:30.746425 29514 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: No bootstrap required, opened a new log
I20240501 19:57:30.746546 29514 ts_tablet_manager.cc:1367] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:57:30.746748 29510 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b: No bootstrap required, opened a new log
I20240501 19:57:30.746836 29510 ts_tablet_manager.cc:1367] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240501 19:57:30.747058 29514 raft_consensus.cc:348] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.747175 29514 raft_consensus.cc:374] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.747198 29510 raft_consensus.cc:348] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.747318 29510 raft_consensus.cc:374] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:57:30.747198 29514 raft_consensus.cc:729] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.747375 29510 raft_consensus.cc:729] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Initialized, Role: FOLLOWER
I20240501 19:57:30.747488 29514 consensus_queue.cc:260] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.747503 29510 consensus_queue.cc:260] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.747802 29510 ts_tablet_manager.cc:1398] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:30.747850 29514 ts_tablet_manager.cc:1398] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:30.768697 29521 raft_consensus.cc:482] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.768900 29521 raft_consensus.cc:504] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.769451 29521 leader_election.cc:288] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.773697 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0518736faac4d54bdb78737484c860a" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:30.773720 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0518736faac4d54bdb78737484c860a" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b" is_pre_election: true
I20240501 19:57:30.773833 29389 raft_consensus.cc:2454] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 0.
I20240501 19:57:30.773865 29464 raft_consensus.cc:2454] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 0.
I20240501 19:57:30.774128 29276 leader_election.cc:302] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.774303 29521 raft_consensus.cc:2781] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.774363 29521 raft_consensus.cc:482] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.774411 29521 raft_consensus.cc:3037] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.775224 29521 raft_consensus.cc:504] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.775542 29521 leader_election.cc:288] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.775808 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0518736faac4d54bdb78737484c860a" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:30.775899 29389 raft_consensus.cc:3037] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.775874 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0518736faac4d54bdb78737484c860a" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:30.775995 29464 raft_consensus.cc:3037] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.776645 29389 raft_consensus.cc:2454] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 1.
I20240501 19:57:30.776698 29464 raft_consensus.cc:2454] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 1.
I20240501 19:57:30.776902 29276 leader_election.cc:302] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.777043 29521 raft_consensus.cc:2781] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.777220 29521 raft_consensus.cc:686] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Becoming Leader. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:30.777336 29521 consensus_queue.cc:237] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.778710 29208 catalog_manager.cc:5484] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 reported cstate change: term changed from 0 to 1, leader changed from <none> to 649b8f2f20324033bc021e229cff5774 (127.25.83.1). New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:30.784896 29521 raft_consensus.cc:482] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.784978 29521 raft_consensus.cc:504] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.785236 29521 leader_election.cc:288] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.785465 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fb3d9c368d4c0d95bf935f60f863e8" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:30.785518 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fb3d9c368d4c0d95bf935f60f863e8" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b" is_pre_election: true
I20240501 19:57:30.785573 29389 raft_consensus.cc:2454] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 0.
I20240501 19:57:30.785629 29464 raft_consensus.cc:2454] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 0.
I20240501 19:57:30.785825 29276 leader_election.cc:302] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.785956 29521 raft_consensus.cc:2781] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.786006 29521 raft_consensus.cc:482] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.786042 29521 raft_consensus.cc:3037] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.786690 29521 raft_consensus.cc:504] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.786947 29521 leader_election.cc:288] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.787182 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fb3d9c368d4c0d95bf935f60f863e8" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:30.787225 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fb3d9c368d4c0d95bf935f60f863e8" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:30.787292 29464 raft_consensus.cc:3037] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.787292 29389 raft_consensus.cc:3037] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.788110 29389 raft_consensus.cc:2454] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 1.
I20240501 19:57:30.788110 29464 raft_consensus.cc:2454] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 1.
I20240501 19:57:30.788400 29278 leader_election.cc:302] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.788554 29521 raft_consensus.cc:2781] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.788668 29521 raft_consensus.cc:686] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Becoming Leader. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:30.788780 29521 consensus_queue.cc:237] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.790153 29208 catalog_manager.cc:5484] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 reported cstate change: term changed from 0 to 1, leader changed from <none> to 649b8f2f20324033bc021e229cff5774 (127.25.83.1). New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:30.791975 29513 raft_consensus.cc:482] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.792048 29513 raft_consensus.cc:504] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.792407 29513 leader_election.cc:288] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.795887 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774" is_pre_election: true
I20240501 19:57:30.795970 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b" is_pre_election: true
I20240501 19:57:30.796006 29313 raft_consensus.cc:2454] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 0.
I20240501 19:57:30.796056 29464 raft_consensus.cc:2454] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 0.
I20240501 19:57:30.796273 29354 leader_election.cc:302] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.796433 29513 raft_consensus.cc:2781] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.796495 29513 raft_consensus.cc:482] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.796545 29513 raft_consensus.cc:3037] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.797237 29513 raft_consensus.cc:504] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.797508 29513 leader_election.cc:288] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.797756 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:30.797756 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:30.797888 29464 raft_consensus.cc:3037] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.797888 29313 raft_consensus.cc:3037] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.798758 29464 raft_consensus.cc:2454] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 1.
I20240501 19:57:30.798758 29313 raft_consensus.cc:2454] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 1.
I20240501 19:57:30.799047 29353 leader_election.cc:302] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.799263 29513 raft_consensus.cc:2781] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.799325 29513 raft_consensus.cc:686] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Becoming Leader. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:30.799430 29513 consensus_queue.cc:237] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.800724 29208 catalog_manager.cc:5484] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2). New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:30.807029 29521 raft_consensus.cc:482] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.807153 29521 raft_consensus.cc:504] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.807485 29521 leader_election.cc:288] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.807777 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:30.807896 29389 raft_consensus.cc:2454] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 0.
I20240501 19:57:30.807992 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b" is_pre_election: true
I20240501 19:57:30.808135 29464 raft_consensus.cc:2454] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 0.
I20240501 19:57:30.808162 29276 leader_election.cc:302] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.808297 29521 raft_consensus.cc:2781] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.808353 29521 raft_consensus.cc:482] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.808406 29521 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.809039 29521 raft_consensus.cc:504] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.809306 29521 leader_election.cc:288] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.809536 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:30.809556 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:30.809623 29389 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.809664 29464 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.810313 29389 raft_consensus.cc:2454] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 1.
I20240501 19:57:30.810320 29464 raft_consensus.cc:2454] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 1.
I20240501 19:57:30.810551 29276 leader_election.cc:302] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.810742 29521 raft_consensus.cc:2781] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.810817 29521 raft_consensus.cc:686] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Becoming Leader. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:30.810932 29521 consensus_queue.cc:237] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.812235 29208 catalog_manager.cc:5484] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 reported cstate change: term changed from 0 to 1, leader changed from <none> to 649b8f2f20324033bc021e229cff5774 (127.25.83.1). New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:30.851521 29521 raft_consensus.cc:482] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.851689 29521 raft_consensus.cc:504] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.852108 29521 leader_election.cc:288] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.853204 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:30.853204 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b" is_pre_election: true
I20240501 19:57:30.853425 29389 raft_consensus.cc:2454] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 0.
I20240501 19:57:30.853430 29464 raft_consensus.cc:2454] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 0.
I20240501 19:57:30.854193 29278 leader_election.cc:302] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.854425 29521 raft_consensus.cc:2781] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.854478 29521 raft_consensus.cc:482] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.854530 29521 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.856283 29521 raft_consensus.cc:504] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.857246 29521 leader_election.cc:288] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 election: Requested vote from peers 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.857860 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:30.857860 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:30.858098 29464 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.858098 29389 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.859272 29389 raft_consensus.cc:2454] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 1.
I20240501 19:57:30.859272 29464 raft_consensus.cc:2454] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 1.
I20240501 19:57:30.859665 29278 leader_election.cc:302] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.859889 29521 raft_consensus.cc:2781] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.859998 29521 raft_consensus.cc:686] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Becoming Leader. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:30.860174 29521 consensus_queue.cc:237] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } }
I20240501 19:57:30.861778 29208 catalog_manager.cc:5484] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 reported cstate change: term changed from 0 to 1, leader changed from <none> to 649b8f2f20324033bc021e229cff5774 (127.25.83.1). New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:30.868218 29513 raft_consensus.cc:482] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.868358 29513 raft_consensus.cc:504] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.868734 29513 leader_election.cc:288] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.869170 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774" is_pre_election: true
I20240501 19:57:30.869184 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b" is_pre_election: true
I20240501 19:57:30.869292 29313 raft_consensus.cc:2454] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 0.
I20240501 19:57:30.869318 29464 raft_consensus.cc:2454] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 0.
I20240501 19:57:30.869520 29353 leader_election.cc:302] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.869658 29513 raft_consensus.cc:2781] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.869704 29513 raft_consensus.cc:482] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.869730 29513 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.870608 29513 raft_consensus.cc:504] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.870887 29513 leader_election.cc:288] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.871217 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:30.871219 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:30.871348 29464 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.871354 29313 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.872051 29313 raft_consensus.cc:2454] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 1.
I20240501 19:57:30.872064 29464 raft_consensus.cc:2454] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 1.
I20240501 19:57:30.872289 29353 leader_election.cc:302] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.872489 29513 raft_consensus.cc:2781] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.872576 29513 raft_consensus.cc:686] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Becoming Leader. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:30.872705 29513 consensus_queue.cc:237] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.874294 29208 catalog_manager.cc:5484] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2). New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:30.879560 29524 raft_consensus.cc:482] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.879647 29524 raft_consensus.cc:504] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.879990 29524 leader_election.cc:288] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.884263 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd1acf0afe14cc2a388fcf79ddae20d" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:30.884286 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd1acf0afe14cc2a388fcf79ddae20d" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774" is_pre_election: true
I20240501 19:57:30.884371 29313 raft_consensus.cc:2454] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:30.884370 29389 raft_consensus.cc:2454] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:30.884640 29428 leader_election.cc:302] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.884792 29524 raft_consensus.cc:2781] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.884848 29524 raft_consensus.cc:482] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.884907 29524 raft_consensus.cc:3037] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.885617 29524 raft_consensus.cc:504] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.885927 29524 leader_election.cc:288] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.886188 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd1acf0afe14cc2a388fcf79ddae20d" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:30.886225 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd1acf0afe14cc2a388fcf79ddae20d" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:30.886284 29313 raft_consensus.cc:3037] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.886299 29389 raft_consensus.cc:3037] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.886919 29389 raft_consensus.cc:2454] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:30.887173 29427 leader_election.cc:302] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.887238 29313 raft_consensus.cc:2454] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:30.887341 29524 raft_consensus.cc:2781] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.887547 29524 raft_consensus.cc:686] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Leader. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:30.887672 29524 consensus_queue.cc:237] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.888975 29208 catalog_manager.cc:5484] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ff839c08d0479993ba43e53980437b (127.25.83.3). New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:30.919503 29524 raft_consensus.cc:482] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.919612 29524 raft_consensus.cc:504] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.920099 29524 leader_election.cc:288] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.920360 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9967547056d848d1b37aff3f6b4abe86" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:30.920348 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9967547056d848d1b37aff3f6b4abe86" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774" is_pre_election: true
I20240501 19:57:30.920491 29313 raft_consensus.cc:2454] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:30.920491 29389 raft_consensus.cc:2454] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:30.920763 29427 leader_election.cc:302] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.920907 29524 raft_consensus.cc:2781] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.920955 29524 raft_consensus.cc:482] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.920985 29524 raft_consensus.cc:3037] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.921659 29524 raft_consensus.cc:504] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.921918 29524 leader_election.cc:288] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.922238 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9967547056d848d1b37aff3f6b4abe86" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:30.922222 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9967547056d848d1b37aff3f6b4abe86" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:30.922343 29313 raft_consensus.cc:3037] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.922363 29389 raft_consensus.cc:3037] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.922935 29313 raft_consensus.cc:2454] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:30.923030 29389 raft_consensus.cc:2454] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:30.923152 29428 leader_election.cc:302] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.923308 29524 raft_consensus.cc:2781] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.923372 29524 raft_consensus.cc:686] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Leader. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:30.923524 29524 consensus_queue.cc:237] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.924840 29208 catalog_manager.cc:5484] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ff839c08d0479993ba43e53980437b (127.25.83.3). New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:30.940054 29524 raft_consensus.cc:482] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.940161 29524 raft_consensus.cc:504] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.940474 29524 leader_election.cc:288] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.940776 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d7f921a55e437eaaddf7d276c8de0c" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774" is_pre_election: true
I20240501 19:57:30.940776 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d7f921a55e437eaaddf7d276c8de0c" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:30.940922 29389 raft_consensus.cc:2454] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:30.940922 29313 raft_consensus.cc:2454] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:30.941176 29428 leader_election.cc:302] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.941318 29524 raft_consensus.cc:2781] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.941373 29524 raft_consensus.cc:482] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.941412 29524 raft_consensus.cc:3037] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.942193 29524 raft_consensus.cc:504] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.942469 29524 leader_election.cc:288] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.942736 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d7f921a55e437eaaddf7d276c8de0c" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:30.942736 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0d7f921a55e437eaaddf7d276c8de0c" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:30.942881 29313 raft_consensus.cc:3037] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.942881 29389 raft_consensus.cc:3037] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.943717 29389 raft_consensus.cc:2454] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:30.943717 29313 raft_consensus.cc:2454] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:30.943991 29427 leader_election.cc:302] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.944144 29524 raft_consensus.cc:2781] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.944223 29524 raft_consensus.cc:686] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Leader. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:30.944353 29524 consensus_queue.cc:237] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.945883 29208 catalog_manager.cc:5484] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ff839c08d0479993ba43e53980437b (127.25.83.3). New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:30.964637 29524 raft_consensus.cc:482] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.964766 29524 raft_consensus.cc:504] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.965121 29524 leader_election.cc:288] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.965409 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:30.965445 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774" is_pre_election: true
I20240501 19:57:30.965523 29389 raft_consensus.cc:2454] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:30.965562 29313 raft_consensus.cc:2454] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:30.965778 29427 leader_election.cc:302] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.965924 29524 raft_consensus.cc:2781] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.965978 29524 raft_consensus.cc:482] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.966018 29524 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.966789 29524 raft_consensus.cc:504] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.967065 29524 leader_election.cc:288] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:30.967324 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:30.967340 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:30.967401 29313 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.967428 29389 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.967998 29313 raft_consensus.cc:2454] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:30.968120 29389 raft_consensus.cc:2454] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:30.968245 29428 leader_election.cc:302] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.968425 29524 raft_consensus.cc:2781] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.968493 29524 raft_consensus.cc:686] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Leader. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:30.968631 29524 consensus_queue.cc:237] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.970113 29208 catalog_manager.cc:5484] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ff839c08d0479993ba43e53980437b (127.25.83.3). New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:30.974718 29513 raft_consensus.cc:482] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.974797 29513 raft_consensus.cc:504] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.975179 29513 leader_election.cc:288] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.975442 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2803395236b144059af4f49d49e8c7e3" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774" is_pre_election: true
I20240501 19:57:30.975476 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2803395236b144059af4f49d49e8c7e3" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b" is_pre_election: true
I20240501 19:57:30.975545 29313 raft_consensus.cc:2454] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 0.
I20240501 19:57:30.975584 29464 raft_consensus.cc:2454] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 0.
I20240501 19:57:30.975805 29354 leader_election.cc:302] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:30.975972 29513 raft_consensus.cc:2781] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:30.976033 29513 raft_consensus.cc:482] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:30.976080 29513 raft_consensus.cc:3037] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.976797 29513 raft_consensus.cc:504] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.977078 29513 leader_election.cc:288] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:30.977285 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2803395236b144059af4f49d49e8c7e3" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:30.977365 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2803395236b144059af4f49d49e8c7e3" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:30.977378 29313 raft_consensus.cc:3037] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.977484 29464 raft_consensus.cc:3037] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:30.978040 29313 raft_consensus.cc:2454] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 1.
I20240501 19:57:30.978209 29464 raft_consensus.cc:2454] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 1.
I20240501 19:57:30.978297 29353 leader_election.cc:302] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:30.978444 29513 raft_consensus.cc:2781] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:30.978526 29513 raft_consensus.cc:686] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Becoming Leader. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:30.978664 29513 consensus_queue.cc:237] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.980049 29208 catalog_manager.cc:5484] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2). New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:30.999356 29524 raft_consensus.cc:482] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:57:30.999445 29524 raft_consensus.cc:504] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:30.999708 29524 leader_election.cc:288] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:31.000030 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb" is_pre_election: true
I20240501 19:57:31.000042 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774" is_pre_election: true
I20240501 19:57:31.000126 29389 raft_consensus.cc:2454] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:31.000169 29313 raft_consensus.cc:2454] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 0.
I20240501 19:57:31.000319 29427 leader_election.cc:302] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:31.000439 29524 raft_consensus.cc:2781] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:57:31.000483 29524 raft_consensus.cc:482] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:57:31.000507 29524 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.001364 29524 raft_consensus.cc:504] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:31.001726 29524 leader_election.cc:288] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:31.002036 29389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:31.002036 29313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:31.002192 29389 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.002192 29313 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:57:31.003015 29313 raft_consensus.cc:2454] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:31.003015 29389 raft_consensus.cc:2454] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 1.
I20240501 19:57:31.003324 29428 leader_election.cc:302] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:31.003465 29524 raft_consensus.cc:2781] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:57:31.003564 29524 raft_consensus.cc:686] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Leader. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:31.003692 29524 consensus_queue.cc:237] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:31.005143 29208 catalog_manager.cc:5484] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b reported cstate change: term changed from 0 to 1, leader changed from <none> to 77ff839c08d0479993ba43e53980437b (127.25.83.3). New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } }
I20240501 19:57:31.061044 29463 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.061044 29464 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.061467 29555 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.061753 29313 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.062319 29516 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.062575 29516 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.062801 29463 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.062956 29389 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.063220 29555 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.066040 29389 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.066695 29461 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.067380 29521 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.069576 29604 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.070132 29521 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.070945 29312 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.071681 29461 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:31.075964 29388 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.076284 29387 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240501 19:57:31.078953 29385 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.079280 29384 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.079494 29521 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.079542 29383 raft_consensus.cc:1261] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.079820 29387 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.080103 29388 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.080998 29312 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.082017 29516 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.082983 29459 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.083082 29461 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.083885 29311 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.083959 29310 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.084192 29309 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.084460 29307 raft_consensus.cc:1261] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.084887 29524 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.084965 29566 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.085170 29524 consensus_queue.cc:1035] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.085682 29312 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:57:31.079790 29521 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.085170 29513 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240501 19:57:31.086202 29516 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.086472 29516 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.086783 29600 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.097345 29566 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.097550 29524 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.097642 29566 consensus_queue.cc:1035] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.097824 29566 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.100664 29581 mvcc.cc:205] Tried to move back new op lower bound from 7022974775555719168 to 7022974774522843136. Current Snapshot: MvccSnapshot[applied={T|T < 7022974775555719168}]
I20240501 19:57:31.102932 29566 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.084435 29607 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.108266 29524 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:57:31.115218 29581 mvcc.cc:205] Tried to move back new op lower bound from 7022974775619346432 to 7022974774984028160. Current Snapshot: MvccSnapshot[applied={T|T < 7022974775591452672}]
I20240501 19:57:31.131989 29610 mvcc.cc:205] Tried to move back new op lower bound from 7022974775619346432 to 7022974774984028160. Current Snapshot: MvccSnapshot[applied={T|T < 7022974775591452672}]
I20240501 19:57:31.496557 29264 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:31.554780 29463 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0"
dest_uuid: "77ff839c08d0479993ba43e53980437b"
mode: GRACEFUL
new_leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:42050
I20240501 19:57:31.555028 29463 raft_consensus.cc:595] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Received request to transfer leadership to TS 26f1ddf6a31645c0a82e697acc9253fb
I20240501 19:57:31.557381 29264 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 1
I20240501 19:57:31.557596 29264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:31.573913 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240501 19:57:31.579016 29634 raft_consensus.cc:979] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b: : Instructing follower 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:31.579120 29634 raft_consensus.cc:1067] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Signalling peer 26f1ddf6a31645c0a82e697acc9253fb to start an election
W20240501 19:57:31.579562 29659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:31.579767 29389 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:40682
W20240501 19:57:31.579946 29661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:31.579957 29389 raft_consensus.cc:482] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:31.580072 29389 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:57:31.580412 29658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:31.580996 29389 raft_consensus.cc:504] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:31.581049 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:31.581248 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:31.581296 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:31.581328 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593451581327 us; error 0 us; skew 500 ppm
I20240501 19:57:31.581543 29389 leader_election.cc:288] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 2 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:31.581631 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:31.581737 29309 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:31.581997 29460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:31.582175 29460 raft_consensus.cc:3032] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:31.582223 29460 raft_consensus.cc:729] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:31.582389 29460 consensus_queue.cc:260] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:31.582511 25932 webserver.cc:460] Webserver started at http://127.25.83.4:41367/ using document root <none> and password file <none>
I20240501 19:57:31.582520 29460 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:31.582691 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:31.582744 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:31.582773 29309 raft_consensus.cc:2454] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 2.
I20240501 19:57:31.582811 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:31.583038 29353 leader_election.cc:302] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:31.583175 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-3-root/instance:
uuid: "b83077a363e241c8b97248c2617f08dc"
format_stamp: "Formatted at 2024-05-01 19:57:31 on dist-test-slave-mfp3"
I20240501 19:57:31.583225 29516 raft_consensus.cc:2781] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:31.583304 29516 raft_consensus.cc:686] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Becoming Leader. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:31.583364 29460 raft_consensus.cc:2454] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 2.
I20240501 19:57:31.583428 29516 consensus_queue.cc:237] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:31.585094 29207 catalog_manager.cc:5484] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: term changed from 1 to 2, leader changed from 77ff839c08d0479993ba43e53980437b (127.25.83.3) to 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2). New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:31.585907 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20240501 19:57:31.586829 29669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.587059 25932 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:31.587128 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-3-root
uuid: "b83077a363e241c8b97248c2617f08dc"
format_stamp: "Formatted at 2024-05-01 19:57:31 on dist-test-slave-mfp3"
I20240501 19:57:31.587190 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:31.593284 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:31.593513 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:31.593715 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:31.594053 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:31.594084 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.594146 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:31.594177 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.598740 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.4:43171
I20240501 19:57:31.598798 29732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.4:43171 every 8 connection(s)
I20240501 19:57:31.599484 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:31.601783 29737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:31.602517 29738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:31.603063 25932 server_base.cc:1034] running on GCE node
W20240501 19:57:31.603592 29740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:31.603816 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:31.603865 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:31.603883 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593451603883 us; error 0 us; skew 500 ppm
I20240501 19:57:31.604076 29733 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34915
I20240501 19:57:31.604172 29733 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:31.604351 29733 heartbeater.cc:511] Master 127.25.83.62:34915 requested a full tablet report, sending...
I20240501 19:57:31.604817 25932 webserver.cc:460] Webserver started at http://127.25.83.5:33213/ using document root <none> and password file <none>
I20240501 19:57:31.604871 29207 ts_manager.cc:194] Registered new tserver with Master: b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171)
I20240501 19:57:31.604947 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:31.604992 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:31.605054 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:31.605396 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-4-root/instance:
uuid: "ba2031c7573846baa1c0190d340bb485"
format_stamp: "Formatted at 2024-05-01 19:57:31 on dist-test-slave-mfp3"
I20240501 19:57:31.605484 29207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56562
I20240501 19:57:31.606830 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:31.607568 29745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.607769 25932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20240501 19:57:31.607827 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-4-root
uuid: "ba2031c7573846baa1c0190d340bb485"
format_stamp: "Formatted at 2024-05-01 19:57:31 on dist-test-slave-mfp3"
I20240501 19:57:31.607900 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:31.616339 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:31.616659 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:31.616947 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:31.617316 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:31.617352 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.617398 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:31.617425 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.621910 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.5:41139
I20240501 19:57:31.621968 29808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.5:41139 every 8 connection(s)
I20240501 19:57:31.622678 25932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:57:31.624816 29812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:31.625385 29813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:57:31.626057 29815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:57:31.626266 29809 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34915
I20240501 19:57:31.626338 29809 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:31.626523 29809 heartbeater.cc:511] Master 127.25.83.62:34915 requested a full tablet report, sending...
I20240501 19:57:31.626646 25932 server_base.cc:1034] running on GCE node
I20240501 19:57:31.626928 25932 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20240501 19:57:31.626979 25932 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20240501 19:57:31.627012 25932 hybrid_clock.cc:648] HybridClock initialized: now 1714593451627011 us; error 0 us; skew 500 ppm
I20240501 19:57:31.627085 29207 ts_manager.cc:194] Registered new tserver with Master: ba2031c7573846baa1c0190d340bb485 (127.25.83.5:41139)
I20240501 19:57:31.627715 29207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56564
I20240501 19:57:31.627887 25932 webserver.cc:460] Webserver started at http://127.25.83.6:34067/ using document root <none> and password file <none>
I20240501 19:57:31.628037 25932 fs_manager.cc:359] Metadata directory not provided
I20240501 19:57:31.628087 25932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:57:31.628155 25932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:57:31.628475 25932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-5-root/instance:
uuid: "cbbe61a596804ca093383d53b3a4b2ab"
format_stamp: "Formatted at 2024-05-01 19:57:31 on dist-test-slave-mfp3"
I20240501 19:57:31.629752 25932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20240501 19:57:31.630522 29820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.630726 25932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.630784 25932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-5-root
uuid: "cbbe61a596804ca093383d53b3a4b2ab"
format_stamp: "Formatted at 2024-05-01 19:57:31 on dist-test-slave-mfp3"
I20240501 19:57:31.630836 25932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:57:31.637226 25932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:57:31.637514 25932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:57:31.637749 25932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:57:31.638172 25932 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:57:31.638211 25932 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.638252 25932 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:57:31.638280 25932 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:57:31.642767 25932 rpc_server.cc:307] RPC server started. Bound to: 127.25.83.6:40237
I20240501 19:57:31.642823 29883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.83.6:40237 every 8 connection(s)
I20240501 19:57:31.646770 29884 heartbeater.cc:347] Connected to a master server at 127.25.83.62:34915
I20240501 19:57:31.646857 29884 heartbeater.cc:464] Registering TS with master...
I20240501 19:57:31.647044 29884 heartbeater.cc:511] Master 127.25.83.62:34915 requested a full tablet report, sending...
I20240501 19:57:31.647572 29207 ts_manager.cc:194] Registered new tserver with Master: cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237)
I20240501 19:57:31.648059 29207 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56570
I20240501 19:57:32.055078 29309 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 19:57:32.055516 29657 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 19:57:32.058951 29460 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20240501 19:57:32.060166 29887 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 19:57:32.505474 29389 consensus_queue.cc:237] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:32.506945 29309 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:32.507395 29458 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:32.507571 29516 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:32.508013 29657 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:32.509174 29516 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:32.509562 29309 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
W20240501 19:57:32.511513 29355 consensus_peers.cc:485] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2803395236b144059af4f49d49e8c7e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:32.511780 29389 consensus_queue.cc:237] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:32.512019 29202 catalog_manager.cc:5484] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index -1 to 29, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:32.512811 29309 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 19:57:32.513314 29891 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:32.513957 29464 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 19:57:32.514194 29458 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:32.514367 29516 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:32.514940 29657 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:32.515345 29309 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
W20240501 19:57:32.517740 29353 consensus_peers.cc:485] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb -> Peer b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171): Couldn't send request to peer b83077a363e241c8b97248c2617f08dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffed8312f8034d9fa6aa423ced8117ee. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:32.518584 29464 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:32.521144 29207 catalog_manager.cc:5484] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index -1 to 28, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:32.521556 29309 consensus_queue.cc:237] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:32.522679 29389 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 19:57:32.522891 29464 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 19:57:32.523048 29603 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:32.523393 29603 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:32.524924 29888 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:32.525211 29389 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:32.525429 29464 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:32.527714 29210 catalog_manager.cc:5484] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index -1 to 28, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:57:32.530238 29277 consensus_peers.cc:485] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 -> Peer ba2031c7573846baa1c0190d340bb485 (127.25.83.5:41139): Couldn't send request to peer ba2031c7573846baa1c0190d340bb485. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0518736faac4d54bdb78737484c860a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:32.532922 29464 consensus_queue.cc:237] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:32.534134 29389 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:32.534294 29309 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:32.534708 29653 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:32.534883 29634 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:32.536280 29634 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:32.536556 29309 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
W20240501 19:57:32.537503 29428 consensus_peers.cc:485] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b -> Peer ba2031c7573846baa1c0190d340bb485 (127.25.83.5:41139): Couldn't send request to peer ba2031c7573846baa1c0190d340bb485. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9967547056d848d1b37aff3f6b4abe86. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:32.538223 29389 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:32.538792 29207 catalog_manager.cc:5484] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index -1 to 29, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:32.539288 29388 consensus_queue.cc:237] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:32.540756 29464 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20240501 19:57:32.540830 29353 consensus_peers.cc:485] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb -> Peer b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171): Couldn't send request to peer b83077a363e241c8b97248c2617f08dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b886671d00534706ad4fc5f6023c1dd0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:32.541108 29309 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 19:57:32.541563 29891 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:32.541994 29516 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:32.543121 29891 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:32.543465 29309 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:32.543704 29464 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:32.545359 29310 consensus_queue.cc:237] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:32.545591 29202 catalog_manager.cc:5484] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index -1 to 28, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:32.547266 29388 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:32.547798 29464 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:32.548126 29603 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:32.548439 29603 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:32.549952 29603 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:32.550392 29388 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
W20240501 19:57:32.550654 29279 consensus_peers.cc:485] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46f49bfbb5624aa89fcdfb1e8eaecc87. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:32.551154 29464 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:32.552546 29207 catalog_manager.cc:5484] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index -1 to 29, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:32.606202 29733 heartbeater.cc:503] Master 127.25.83.62:34915 was elected leader, sending a full tablet report...
I20240501 19:57:32.628474 29809 heartbeater.cc:503] Master 127.25.83.62:34915 was elected leader, sending a full tablet report...
I20240501 19:57:32.648859 29884 heartbeater.cc:503] Master 127.25.83.62:34915 was elected leader, sending a full tablet report...
I20240501 19:57:32.895352 29907 ts_tablet_manager.cc:902] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: Initiating tablet copy from peer 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:32.895804 29907 tablet_copy_client.cc:308] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.2:32945
I20240501 19:57:32.899247 29399 tablet_copy_service.cc:140] P 26f1ddf6a31645c0a82e697acc9253fb: Received BeginTabletCopySession request for tablet ffed8312f8034d9fa6aa423ced8117ee from peer b83077a363e241c8b97248c2617f08dc ({username='slave'} at 127.0.0.1:40718)
I20240501 19:57:32.899346 29399 tablet_copy_service.cc:161] P 26f1ddf6a31645c0a82e697acc9253fb: Beginning new tablet copy session on tablet ffed8312f8034d9fa6aa423ced8117ee from peer b83077a363e241c8b97248c2617f08dc at {username='slave'} at 127.0.0.1:40718: session id = b83077a363e241c8b97248c2617f08dc-ffed8312f8034d9fa6aa423ced8117ee
I20240501 19:57:32.900256 29399 tablet_copy_source_session.cc:205] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:32.900964 29907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffed8312f8034d9fa6aa423ced8117ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.902714 29907 tablet_copy_client.cc:783] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:32.902864 29907 tablet_copy_client.cc:647] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:32.903725 29911 tablet_copy_client.cc:1002] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:32.954701 29913 ts_tablet_manager.cc:902] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: Initiating tablet copy from peer 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:32.955309 29913 tablet_copy_client.cc:308] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.2:32945
I20240501 19:57:32.955678 29399 tablet_copy_service.cc:140] P 26f1ddf6a31645c0a82e697acc9253fb: Received BeginTabletCopySession request for tablet b886671d00534706ad4fc5f6023c1dd0 from peer b83077a363e241c8b97248c2617f08dc ({username='slave'} at 127.0.0.1:40718)
I20240501 19:57:32.955780 29399 tablet_copy_service.cc:161] P 26f1ddf6a31645c0a82e697acc9253fb: Beginning new tablet copy session on tablet b886671d00534706ad4fc5f6023c1dd0 from peer b83077a363e241c8b97248c2617f08dc at {username='slave'} at 127.0.0.1:40718: session id = b83077a363e241c8b97248c2617f08dc-b886671d00534706ad4fc5f6023c1dd0
I20240501 19:57:32.956795 29399 tablet_copy_source_session.cc:205] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:32.957532 29913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b886671d00534706ad4fc5f6023c1dd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:32.959656 29913 tablet_copy_client.cc:783] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:32.959823 29913 tablet_copy_client.cc:647] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:32.960757 29917 tablet_copy_client.cc:1002] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:33.008345 29918 ts_tablet_manager.cc:902] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: Initiating tablet copy from peer 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115)
I20240501 19:57:33.008962 29918 tablet_copy_client.cc:308] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.1:46115
I20240501 19:57:33.012727 29323 tablet_copy_service.cc:140] P 649b8f2f20324033bc021e229cff5774: Received BeginTabletCopySession request for tablet e0518736faac4d54bdb78737484c860a from peer ba2031c7573846baa1c0190d340bb485 ({username='slave'} at 127.0.0.1:40596)
I20240501 19:57:33.012831 29323 tablet_copy_service.cc:161] P 649b8f2f20324033bc021e229cff5774: Beginning new tablet copy session on tablet e0518736faac4d54bdb78737484c860a from peer ba2031c7573846baa1c0190d340bb485 at {username='slave'} at 127.0.0.1:40596: session id = ba2031c7573846baa1c0190d340bb485-e0518736faac4d54bdb78737484c860a
I20240501 19:57:33.014223 29323 tablet_copy_source_session.cc:205] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:33.015077 29918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0518736faac4d54bdb78737484c860a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.017136 29918 tablet_copy_client.cc:783] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:33.017328 29918 tablet_copy_client.cc:647] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:33.018182 29922 tablet_copy_client.cc:1002] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:33.059338 29924 ts_tablet_manager.cc:902] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: Initiating tablet copy from peer 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:33.059935 29924 tablet_copy_client.cc:308] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.3:44839
I20240501 19:57:33.063490 29474 tablet_copy_service.cc:140] P 77ff839c08d0479993ba43e53980437b: Received BeginTabletCopySession request for tablet 9967547056d848d1b37aff3f6b4abe86 from peer ba2031c7573846baa1c0190d340bb485 ({username='slave'} at 127.0.0.1:42058)
I20240501 19:57:33.063589 29474 tablet_copy_service.cc:161] P 77ff839c08d0479993ba43e53980437b: Beginning new tablet copy session on tablet 9967547056d848d1b37aff3f6b4abe86 from peer ba2031c7573846baa1c0190d340bb485 at {username='slave'} at 127.0.0.1:42058: session id = ba2031c7573846baa1c0190d340bb485-9967547056d848d1b37aff3f6b4abe86
I20240501 19:57:33.064486 29474 tablet_copy_source_session.cc:205] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:33.065575 29924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9967547056d848d1b37aff3f6b4abe86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.067710 29924 tablet_copy_client.cc:783] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:33.067917 29924 tablet_copy_client.cc:647] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:33.068989 29926 tablet_copy_client.cc:1002] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:33.100466 29930 ts_tablet_manager.cc:902] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: Initiating tablet copy from peer 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:33.101085 29930 tablet_copy_client.cc:308] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.2:32945
I20240501 19:57:33.104362 29399 tablet_copy_service.cc:140] P 26f1ddf6a31645c0a82e697acc9253fb: Received BeginTabletCopySession request for tablet 2803395236b144059af4f49d49e8c7e3 from peer cbbe61a596804ca093383d53b3a4b2ab ({username='slave'} at 127.0.0.1:40720)
I20240501 19:57:33.104460 29399 tablet_copy_service.cc:161] P 26f1ddf6a31645c0a82e697acc9253fb: Beginning new tablet copy session on tablet 2803395236b144059af4f49d49e8c7e3 from peer cbbe61a596804ca093383d53b3a4b2ab at {username='slave'} at 127.0.0.1:40720: session id = cbbe61a596804ca093383d53b3a4b2ab-2803395236b144059af4f49d49e8c7e3
I20240501 19:57:33.105393 29399 tablet_copy_source_session.cc:205] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:33.106065 29930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2803395236b144059af4f49d49e8c7e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.107981 29930 tablet_copy_client.cc:783] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:33.108162 29930 tablet_copy_client.cc:647] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:33.109068 29934 tablet_copy_client.cc:1002] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:33.109776 29936 ts_tablet_manager.cc:902] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: Initiating tablet copy from peer 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115)
I20240501 19:57:33.110277 29936 tablet_copy_client.cc:308] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.1:46115
I20240501 19:57:33.113341 29323 tablet_copy_service.cc:140] P 649b8f2f20324033bc021e229cff5774: Received BeginTabletCopySession request for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 from peer cbbe61a596804ca093383d53b3a4b2ab ({username='slave'} at 127.0.0.1:40602)
I20240501 19:57:33.113428 29323 tablet_copy_service.cc:161] P 649b8f2f20324033bc021e229cff5774: Beginning new tablet copy session on tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 from peer cbbe61a596804ca093383d53b3a4b2ab at {username='slave'} at 127.0.0.1:40602: session id = cbbe61a596804ca093383d53b3a4b2ab-46f49bfbb5624aa89fcdfb1e8eaecc87
I20240501 19:57:33.114147 29323 tablet_copy_source_session.cc:205] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:33.114732 29936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:33.116648 29936 tablet_copy_client.cc:783] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:33.116815 29936 tablet_copy_client.cc:647] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:33.117638 29940 tablet_copy_client.cc:1002] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:33.557837 29264 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:33.558984 29264 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:33.559067 29264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:34.904017 29907 tablet_copy_client.cc:523] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:34.905465 29907 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: Bootstrap starting.
I20240501 19:57:34.927186 29907 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:34.927439 29907 tablet_bootstrap.cc:492] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: Bootstrap complete.
I20240501 19:57:34.927603 29907 ts_tablet_manager.cc:1367] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.007s
I20240501 19:57:34.928020 29907 raft_consensus.cc:348] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:34.928160 29907 raft_consensus.cc:729] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b83077a363e241c8b97248c2617f08dc, State: Initialized, Role: LEARNER
I20240501 19:57:34.928256 29907 consensus_queue.cc:260] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:34.928758 29907 ts_tablet_manager.cc:1398] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:34.929167 29399 tablet_copy_service.cc:342] P 26f1ddf6a31645c0a82e697acc9253fb: Request end of tablet copy session b83077a363e241c8b97248c2617f08dc-ffed8312f8034d9fa6aa423ced8117ee received from {username='slave'} at 127.0.0.1:40718
I20240501 19:57:34.929235 29399 tablet_copy_service.cc:433] P 26f1ddf6a31645c0a82e697acc9253fb: ending tablet copy session b83077a363e241c8b97248c2617f08dc-ffed8312f8034d9fa6aa423ced8117ee on tablet ffed8312f8034d9fa6aa423ced8117ee with peer b83077a363e241c8b97248c2617f08dc
I20240501 19:57:34.961023 29913 tablet_copy_client.cc:523] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:34.962450 29913 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: Bootstrap starting.
I20240501 19:57:34.982535 29913 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:34.982797 29913 tablet_bootstrap.cc:492] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: Bootstrap complete.
I20240501 19:57:34.982962 29913 ts_tablet_manager.cc:1367] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.007s
I20240501 19:57:34.983379 29913 raft_consensus.cc:348] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:34.983506 29913 raft_consensus.cc:729] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b83077a363e241c8b97248c2617f08dc, State: Initialized, Role: LEARNER
I20240501 19:57:34.983608 29913 consensus_queue.cc:260] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:34.983934 29913 ts_tablet_manager.cc:1398] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:34.984323 29399 tablet_copy_service.cc:342] P 26f1ddf6a31645c0a82e697acc9253fb: Request end of tablet copy session b83077a363e241c8b97248c2617f08dc-b886671d00534706ad4fc5f6023c1dd0 received from {username='slave'} at 127.0.0.1:40718
I20240501 19:57:34.984401 29399 tablet_copy_service.cc:433] P 26f1ddf6a31645c0a82e697acc9253fb: ending tablet copy session b83077a363e241c8b97248c2617f08dc-b886671d00534706ad4fc5f6023c1dd0 on tablet b886671d00534706ad4fc5f6023c1dd0 with peer b83077a363e241c8b97248c2617f08dc
I20240501 19:57:35.018435 29918 tablet_copy_client.cc:523] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:35.019807 29918 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: Bootstrap starting.
I20240501 19:57:35.020504 29707 raft_consensus.cc:1203] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240501 19:57:35.041750 29918 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:35.042060 29918 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: Bootstrap complete.
I20240501 19:57:35.042279 29918 ts_tablet_manager.cc:1367] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20240501 19:57:35.042747 29918 raft_consensus.cc:348] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:35.042868 29918 raft_consensus.cc:729] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba2031c7573846baa1c0190d340bb485, State: Initialized, Role: LEARNER
I20240501 19:57:35.042968 29918 consensus_queue.cc:260] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:35.043330 29918 ts_tablet_manager.cc:1398] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:35.043861 29323 tablet_copy_service.cc:342] P 649b8f2f20324033bc021e229cff5774: Request end of tablet copy session ba2031c7573846baa1c0190d340bb485-e0518736faac4d54bdb78737484c860a received from {username='slave'} at 127.0.0.1:40596
I20240501 19:57:35.043967 29323 tablet_copy_service.cc:433] P 649b8f2f20324033bc021e229cff5774: ending tablet copy session ba2031c7573846baa1c0190d340bb485-e0518736faac4d54bdb78737484c860a on tablet e0518736faac4d54bdb78737484c860a with peer ba2031c7573846baa1c0190d340bb485
I20240501 19:57:35.057674 29783 raft_consensus.cc:1203] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240501 19:57:35.069280 29924 tablet_copy_client.cc:523] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:35.070652 29924 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: Bootstrap starting.
I20240501 19:57:35.091863 29924 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:35.092176 29924 tablet_bootstrap.cc:492] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: Bootstrap complete.
I20240501 19:57:35.092365 29924 ts_tablet_manager.cc:1367] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: Time spent bootstrapping tablet: real 0.022s user 0.017s sys 0.003s
I20240501 19:57:35.092862 29924 raft_consensus.cc:348] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:35.092972 29924 raft_consensus.cc:729] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba2031c7573846baa1c0190d340bb485, State: Initialized, Role: LEARNER
I20240501 19:57:35.093087 29924 consensus_queue.cc:260] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:35.093360 29924 ts_tablet_manager.cc:1398] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:35.093742 29474 tablet_copy_service.cc:342] P 77ff839c08d0479993ba43e53980437b: Request end of tablet copy session ba2031c7573846baa1c0190d340bb485-9967547056d848d1b37aff3f6b4abe86 received from {username='slave'} at 127.0.0.1:42058
I20240501 19:57:35.093812 29474 tablet_copy_service.cc:433] P 77ff839c08d0479993ba43e53980437b: ending tablet copy session ba2031c7573846baa1c0190d340bb485-9967547056d848d1b37aff3f6b4abe86 on tablet 9967547056d848d1b37aff3f6b4abe86 with peer ba2031c7573846baa1c0190d340bb485
I20240501 19:57:35.109316 29930 tablet_copy_client.cc:523] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:35.110666 29930 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap starting.
I20240501 19:57:35.117873 29936 tablet_copy_client.cc:523] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:35.119112 29936 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap starting.
I20240501 19:57:35.133653 29930 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:35.133905 29930 tablet_bootstrap.cc:492] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap complete.
I20240501 19:57:35.134076 29930 ts_tablet_manager.cc:1367] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.001s
I20240501 19:57:35.134496 29930 raft_consensus.cc:348] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:35.134599 29930 raft_consensus.cc:729] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbbe61a596804ca093383d53b3a4b2ab, State: Initialized, Role: LEARNER
I20240501 19:57:35.134692 29930 consensus_queue.cc:260] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:35.134994 29930 ts_tablet_manager.cc:1398] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:35.135701 29399 tablet_copy_service.cc:342] P 26f1ddf6a31645c0a82e697acc9253fb: Request end of tablet copy session cbbe61a596804ca093383d53b3a4b2ab-2803395236b144059af4f49d49e8c7e3 received from {username='slave'} at 127.0.0.1:40720
I20240501 19:57:35.135784 29399 tablet_copy_service.cc:433] P 26f1ddf6a31645c0a82e697acc9253fb: ending tablet copy session cbbe61a596804ca093383d53b3a4b2ab-2803395236b144059af4f49d49e8c7e3 on tablet 2803395236b144059af4f49d49e8c7e3 with peer cbbe61a596804ca093383d53b3a4b2ab
I20240501 19:57:35.143267 29783 raft_consensus.cc:1203] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 19:57:35.146972 29936 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=127 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:35.147298 29936 tablet_bootstrap.cc:492] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap complete.
I20240501 19:57:35.147542 29936 ts_tablet_manager.cc:1367] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.005s
I20240501 19:57:35.149312 29936 raft_consensus.cc:348] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:35.149538 29936 raft_consensus.cc:729] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbbe61a596804ca093383d53b3a4b2ab, State: Initialized, Role: LEARNER
I20240501 19:57:35.149681 29936 consensus_queue.cc:260] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:35.150038 29936 ts_tablet_manager.cc:1398] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 19:57:35.150719 29323 tablet_copy_service.cc:342] P 649b8f2f20324033bc021e229cff5774: Request end of tablet copy session cbbe61a596804ca093383d53b3a4b2ab-46f49bfbb5624aa89fcdfb1e8eaecc87 received from {username='slave'} at 127.0.0.1:40602
I20240501 19:57:35.151273 29323 tablet_copy_service.cc:433] P 649b8f2f20324033bc021e229cff5774: ending tablet copy session cbbe61a596804ca093383d53b3a4b2ab-46f49bfbb5624aa89fcdfb1e8eaecc87 on tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 with peer cbbe61a596804ca093383d53b3a4b2ab
I20240501 19:57:35.159396 29858 raft_consensus.cc:1203] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 19:57:35.273286 29858 raft_consensus.cc:1203] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 19:57:35.333210 29707 raft_consensus.cc:1203] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20240501 19:57:35.556303 29941 raft_consensus.cc:1050] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb: attempting to promote NON_VOTER b83077a363e241c8b97248c2617f08dc to VOTER
I20240501 19:57:35.556747 29941 consensus_queue.cc:237] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:35.559089 29464 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:35.559316 29264 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:35.559981 29959 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.560144 29310 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:35.560837 29264 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 0
I20240501 19:57:35.560969 29264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:35.561178 29707 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:35.561175 29959 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.561775 29941 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.563905 29962 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.564311 29707 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.564282 29309 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.566175 29464 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.566902 29210 catalog_manager.cc:5484] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:35.571383 29388 consensus_queue.cc:237] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:35.572381 29464 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.572414 29707 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.572798 29961 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.573014 29941 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.574765 29962 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.575006 29707 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.575021 29464 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.577066 29194 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffed8312f8034d9fa6aa423ced8117ee with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:35.577399 29202 catalog_manager.cc:5484] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index 29 to 30, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.579567 29300 tablet_service.cc:1514] Processing DeleteTablet for tablet ffed8312f8034d9fa6aa423ced8117ee with delete_type TABLET_DATA_TOMBSTONED (TS 649b8f2f20324033bc021e229cff5774 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:35.579926 29970 tablet_replica.cc:330] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: stopping tablet replica
I20240501 19:57:35.580310 29970 raft_consensus.cc:2229] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:35.580531 29970 raft_consensus.cc:2258] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:35.581332 29970 ts_tablet_manager.cc:1875] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:35.583549 29970 ts_tablet_manager.cc:1888] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240501 19:57:35.583619 29970 log.cc:1196] T ffed8312f8034d9fa6aa423ced8117ee P 649b8f2f20324033bc021e229cff5774: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root/wals/ffed8312f8034d9fa6aa423ced8117ee
I20240501 19:57:35.584100 29195 catalog_manager.cc:4830] TS 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115): tablet ffed8312f8034d9fa6aa423ced8117ee (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:35.594269 29972 raft_consensus.cc:1050] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: attempting to promote NON_VOTER ba2031c7573846baa1c0190d340bb485 to VOTER
I20240501 19:57:35.594677 29972 consensus_queue.cc:237] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:35.596297 29464 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:35.596583 29783 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:35.596623 29971 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.596905 29974 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.596966 29388 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:35.597786 29974 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.599375 29972 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.599741 29783 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.601095 29464 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.601513 29389 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.602077 29205 catalog_manager.cc:5484] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 reported cstate change: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.605384 29309 consensus_queue.cc:237] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:35.606442 29783 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.606879 29972 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.607327 29464 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.607631 29972 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.608319 29954 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.609508 29464 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.609827 29195 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0518736faac4d54bdb78737484c860a with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:35.610239 29783 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.610452 29207 catalog_manager.cc:5484] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index 29 to 30, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:35.612546 29376 tablet_service.cc:1514] Processing DeleteTablet for tablet e0518736faac4d54bdb78737484c860a with delete_type TABLET_DATA_TOMBSTONED (TS 26f1ddf6a31645c0a82e697acc9253fb not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:40664
I20240501 19:57:35.613687 29978 tablet_replica.cc:330] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: stopping tablet replica
I20240501 19:57:35.613919 29978 raft_consensus.cc:2229] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:35.614085 29978 raft_consensus.cc:2258] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:35.614684 29978 ts_tablet_manager.cc:1875] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:35.616638 29978 ts_tablet_manager.cc:1888] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240501 19:57:35.616706 29978 log.cc:1196] T e0518736faac4d54bdb78737484c860a P 26f1ddf6a31645c0a82e697acc9253fb: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root/wals/e0518736faac4d54bdb78737484c860a
I20240501 19:57:35.617174 29194 catalog_manager.cc:4830] TS 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945): tablet e0518736faac4d54bdb78737484c860a (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:35.620601 29959 raft_consensus.cc:1050] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb: attempting to promote NON_VOTER cbbe61a596804ca093383d53b3a4b2ab to VOTER
I20240501 19:57:35.620931 29959 consensus_queue.cc:237] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:35.622078 29858 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.622190 29309 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.622390 29464 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.622476 29961 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.622785 29961 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.622964 29941 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.624455 29941 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.624857 29458 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.626060 29309 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.626772 29207 catalog_manager.cc:5484] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.628569 29858 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.631431 29388 consensus_queue.cc:237] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:35.632282 29858 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:35.632313 29309 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:35.632627 29959 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:35.632810 29961 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:35.634173 29959 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.634680 29858 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.635735 29194 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2803395236b144059af4f49d49e8c7e3 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:35.635689 29309 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.636312 29202 catalog_manager.cc:5484] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index 30 to 31, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:35.639299 29454 tablet_service.cc:1514] Processing DeleteTablet for tablet 2803395236b144059af4f49d49e8c7e3 with delete_type TABLET_DATA_TOMBSTONED (TS 77ff839c08d0479993ba43e53980437b not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:42002
I20240501 19:57:35.639623 29984 tablet_replica.cc:330] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: stopping tablet replica
I20240501 19:57:35.639778 29984 raft_consensus.cc:2229] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:35.639981 29984 raft_consensus.cc:2258] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:35.640584 29984 ts_tablet_manager.cc:1875] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:35.642467 29984 ts_tablet_manager.cc:1888] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 19:57:35.642527 29984 log.cc:1196] T 2803395236b144059af4f49d49e8c7e3 P 77ff839c08d0479993ba43e53980437b: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root/wals/2803395236b144059af4f49d49e8c7e3
I20240501 19:57:35.643021 29196 catalog_manager.cc:4830] TS 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839): tablet 2803395236b144059af4f49d49e8c7e3 (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:35.674872 29971 raft_consensus.cc:1050] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774: attempting to promote NON_VOTER cbbe61a596804ca093383d53b3a4b2ab to VOTER
I20240501 19:57:35.675278 29971 consensus_queue.cc:237] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:35.676438 29858 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.676512 29387 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.676484 29458 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.676862 29954 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.677038 29972 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.677213 29974 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.679332 29971 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.679760 29458 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.681767 29210 catalog_manager.cc:5484] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.681842 29387 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.683174 29858 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.684912 29309 consensus_queue.cc:237] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:35.685971 29387 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:35.686084 29858 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:35.686345 29974 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:35.686503 29954 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:35.687955 29972 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.689150 29195 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:35.689519 29387 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.689595 29858 raft_consensus.cc:2932] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:35.690009 29202 catalog_manager.cc:5484] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index 30 to 31, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:35.692481 29454 tablet_service.cc:1514] Processing DeleteTablet for tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 with delete_type TABLET_DATA_TOMBSTONED (TS 77ff839c08d0479993ba43e53980437b not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:42002
I20240501 19:57:35.692677 29984 tablet_replica.cc:330] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: stopping tablet replica
I20240501 19:57:35.692811 29984 raft_consensus.cc:2229] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:35.692905 29984 raft_consensus.cc:2258] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:35.693555 29984 ts_tablet_manager.cc:1875] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:35.695780 29984 ts_tablet_manager.cc:1888] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 19:57:35.695847 29984 log.cc:1196] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 77ff839c08d0479993ba43e53980437b: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root/wals/46f49bfbb5624aa89fcdfb1e8eaecc87
I20240501 19:57:35.696364 29196 catalog_manager.cc:4830] TS 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839): tablet 46f49bfbb5624aa89fcdfb1e8eaecc87 (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:35.760488 29989 raft_consensus.cc:1050] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b: attempting to promote NON_VOTER ba2031c7573846baa1c0190d340bb485 to VOTER
I20240501 19:57:35.761153 29989 consensus_queue.cc:237] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:35.762501 29783 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.762517 29309 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.762517 29387 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:35.762928 29948 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.763113 29989 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.763633 29991 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.765561 29989 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.765884 29309 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.767355 29387 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.767596 29783 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.768085 29210 catalog_manager.cc:5484] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:35.771070 29464 consensus_queue.cc:237] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:35.772037 29309 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:35.772101 29783 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:35.772403 29948 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:35.772727 29991 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:35.773988 29979 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.774412 29783 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.774626 29309 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.775516 29196 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9967547056d848d1b37aff3f6b4abe86 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:35.776202 29205 catalog_manager.cc:5484] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 reported cstate change: config changed from index 30 to 31, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:35.778221 29376 tablet_service.cc:1514] Processing DeleteTablet for tablet 9967547056d848d1b37aff3f6b4abe86 with delete_type TABLET_DATA_TOMBSTONED (TS 26f1ddf6a31645c0a82e697acc9253fb not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:40664
I20240501 19:57:35.778415 29978 tablet_replica.cc:330] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: stopping tablet replica
I20240501 19:57:35.778548 29978 raft_consensus.cc:2229] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:35.778712 29978 raft_consensus.cc:2258] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:35.779256 29978 ts_tablet_manager.cc:1875] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:35.780871 29978 ts_tablet_manager.cc:1888] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 19:57:35.780947 29978 log.cc:1196] T 9967547056d848d1b37aff3f6b4abe86 P 26f1ddf6a31645c0a82e697acc9253fb: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root/wals/9967547056d848d1b37aff3f6b4abe86
I20240501 19:57:35.781337 29194 catalog_manager.cc:4830] TS 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945): tablet 9967547056d848d1b37aff3f6b4abe86 (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:35.805692 29959 raft_consensus.cc:1050] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb: attempting to promote NON_VOTER b83077a363e241c8b97248c2617f08dc to VOTER
I20240501 19:57:35.806038 29959 consensus_queue.cc:237] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:35.807261 29707 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 LEARNER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 19:57:35.807410 29464 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 19:57:35.807592 29941 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.807410 29309 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 19:57:35.807933 29941 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.808049 29962 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:35.809732 29962 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.810096 29309 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.810457 29707 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.811539 29464 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.812557 29202 catalog_manager.cc:5484] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index 28 to 29, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.815572 29388 consensus_queue.cc:237] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:35.816428 29464 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 19:57:35.816439 29707 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 19:57:35.816759 29941 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.816897 29959 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:35.818158 29941 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.818554 29464 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.819769 29194 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b886671d00534706ad4fc5f6023c1dd0 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:35.820075 29205 catalog_manager.cc:5484] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index 29 to 30, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:35.820817 29707 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:35.821995 29300 tablet_service.cc:1514] Processing DeleteTablet for tablet b886671d00534706ad4fc5f6023c1dd0 with delete_type TABLET_DATA_TOMBSTONED (TS 649b8f2f20324033bc021e229cff5774 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:35.822190 29970 tablet_replica.cc:330] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: stopping tablet replica
I20240501 19:57:35.822321 29970 raft_consensus.cc:2229] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:35.822464 29970 raft_consensus.cc:2258] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:35.823186 29970 ts_tablet_manager.cc:1875] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:35.825119 29970 ts_tablet_manager.cc:1888] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20240501 19:57:35.825191 29970 log.cc:1196] T b886671d00534706ad4fc5f6023c1dd0 P 649b8f2f20324033bc021e229cff5774: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root/wals/b886671d00534706ad4fc5f6023c1dd0
I20240501 19:57:35.825646 29195 catalog_manager.cc:4830] TS 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115): tablet b886671d00534706ad4fc5f6023c1dd0 (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:36.823664 29464 consensus_queue.cc:237] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:36.825130 29387 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 19:57:36.825213 29309 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20240501 19:57:36.825557 29989 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 19:57:36.825748 30001 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20240501 19:57:36.827351 30000 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:36.828007 29387 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
W20240501 19:57:36.830287 29430 consensus_peers.cc:485] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2f89b5db8c34edaa8751a263dc818cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:36.830353 29207 catalog_manager.cc:5484] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index -1 to 27, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:36.830502 29310 consensus_queue.cc:237] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:36.830891 29309 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:36.831686 29387 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:36.832015 29972 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:36.832710 29464 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
W20240501 19:57:36.832793 29277 consensus_peers.cc:485] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 -> Peer ba2031c7573846baa1c0190d340bb485 (127.25.83.5:41139): Couldn't send request to peer ba2031c7573846baa1c0190d340bb485. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39fb3d9c368d4c0d95bf935f60f863e8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:36.833089 30008 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:36.833798 30008 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:36.836261 29464 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:36.836431 29309 consensus_queue.cc:237] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:36.836645 29210 catalog_manager.cc:5484] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index -1 to 29, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:36.837744 29388 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:36.838303 29464 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:36.838551 30009 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:36.838704 29972 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:36.838712 29387 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:36.841168 30015 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:36.841503 29387 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
W20240501 19:57:36.842749 29277 consensus_peers.cc:485] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 -> Peer b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171): Couldn't send request to peer b83077a363e241c8b97248c2617f08dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fe56fe5bfe2443c97aa252754a24aa6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:36.844177 29309 consensus_queue.cc:237] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:36.846292 29210 catalog_manager.cc:5484] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index -1 to 29, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:36.846928 29783 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
W20240501 19:57:36.847540 29277 consensus_peers.cc:485] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 -> Peer b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171): Couldn't send request to peer b83077a363e241c8b97248c2617f08dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0518736faac4d54bdb78737484c860a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:36.847589 30015 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:36.848862 29464 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:36.849649 29458 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index -1 to 29, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:36.850637 30015 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:36.851758 30008 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:36.853793 29783 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:36.854462 29207 catalog_manager.cc:5484] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index 30 to 31, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:36.854857 29464 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:36.855701 29458 consensus_queue.cc:237] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:36.857443 29309 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 19:57:36.857487 29387 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 19:57:36.858167 29989 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20240501 19:57:36.858354 29428 consensus_peers.cc:485] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b -> Peer ba2031c7573846baa1c0190d340bb485 (127.25.83.5:41139): Couldn't send request to peer ba2031c7573846baa1c0190d340bb485. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0d7f921a55e437eaaddf7d276c8de0c. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:36.858523 29989 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:36.860061 30000 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:36.860435 29309 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:36.860453 29387 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:36.862363 29388 consensus_queue.cc:237] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:36.862560 29210 catalog_manager.cc:5484] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index -1 to 28, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:36.863971 29309 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240501 19:57:36.864437 29959 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:36.864540 29458 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
W20240501 19:57:36.864614 29355 consensus_peers.cc:485] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e50e51f9e3c4d3d83ae49f7a326551a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:36.865072 29959 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:36.866568 30029 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:36.866986 29309 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:36.867283 29458 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:36.869391 29205 catalog_manager.cc:5484] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index -1 to 28, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New cstate: current_term: 1 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:37.217100 30033 ts_tablet_manager.cc:902] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: Initiating tablet copy from peer 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115)
I20240501 19:57:37.217721 30033 tablet_copy_client.cc:308] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.1:46115
I20240501 19:57:37.218124 29323 tablet_copy_service.cc:140] P 649b8f2f20324033bc021e229cff5774: Received BeginTabletCopySession request for tablet 39fb3d9c368d4c0d95bf935f60f863e8 from peer ba2031c7573846baa1c0190d340bb485 ({username='slave'} at 127.0.0.1:40596)
I20240501 19:57:37.218209 29323 tablet_copy_service.cc:161] P 649b8f2f20324033bc021e229cff5774: Beginning new tablet copy session on tablet 39fb3d9c368d4c0d95bf935f60f863e8 from peer ba2031c7573846baa1c0190d340bb485 at {username='slave'} at 127.0.0.1:40596: session id = ba2031c7573846baa1c0190d340bb485-39fb3d9c368d4c0d95bf935f60f863e8
I20240501 19:57:37.218971 29323 tablet_copy_source_session.cc:205] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:37.219715 30033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39fb3d9c368d4c0d95bf935f60f863e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.221753 30033 tablet_copy_client.cc:783] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:37.221908 30033 tablet_copy_client.cc:647] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:37.222985 30037 tablet_copy_client.cc:1002] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:37.271900 30038 ts_tablet_manager.cc:902] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: Initiating tablet copy from peer 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115)
I20240501 19:57:37.272544 30038 tablet_copy_client.cc:308] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.1:46115
I20240501 19:57:37.276826 29323 tablet_copy_service.cc:140] P 649b8f2f20324033bc021e229cff5774: Received BeginTabletCopySession request for tablet 5fe56fe5bfe2443c97aa252754a24aa6 from peer b83077a363e241c8b97248c2617f08dc ({username='slave'} at 127.0.0.1:40612)
I20240501 19:57:37.276942 29323 tablet_copy_service.cc:161] P 649b8f2f20324033bc021e229cff5774: Beginning new tablet copy session on tablet 5fe56fe5bfe2443c97aa252754a24aa6 from peer b83077a363e241c8b97248c2617f08dc at {username='slave'} at 127.0.0.1:40612: session id = b83077a363e241c8b97248c2617f08dc-5fe56fe5bfe2443c97aa252754a24aa6
I20240501 19:57:37.277091 30045 ts_tablet_manager.cc:902] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: Initiating tablet copy from peer 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:37.277585 30045 tablet_copy_client.cc:308] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.3:44839
I20240501 19:57:37.277894 29323 tablet_copy_source_session.cc:205] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:37.278684 30038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fe56fe5bfe2443c97aa252754a24aa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.280758 30038 tablet_copy_client.cc:783] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:37.280917 30038 tablet_copy_client.cc:647] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:37.281626 29474 tablet_copy_service.cc:140] P 77ff839c08d0479993ba43e53980437b: Received BeginTabletCopySession request for tablet a2f89b5db8c34edaa8751a263dc818cd from peer cbbe61a596804ca093383d53b3a4b2ab ({username='slave'} at 127.0.0.1:42064)
I20240501 19:57:37.281715 29474 tablet_copy_service.cc:161] P 77ff839c08d0479993ba43e53980437b: Beginning new tablet copy session on tablet a2f89b5db8c34edaa8751a263dc818cd from peer cbbe61a596804ca093383d53b3a4b2ab at {username='slave'} at 127.0.0.1:42064: session id = cbbe61a596804ca093383d53b3a4b2ab-a2f89b5db8c34edaa8751a263dc818cd
I20240501 19:57:37.282478 29474 tablet_copy_source_session.cc:205] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:37.282554 30042 tablet_copy_client.cc:1002] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:37.283102 30045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f89b5db8c34edaa8751a263dc818cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.284710 30045 tablet_copy_client.cc:783] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:37.284857 30045 tablet_copy_client.cc:647] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:37.285894 30049 tablet_copy_client.cc:1002] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:37.406755 30052 ts_tablet_manager.cc:902] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: Initiating tablet copy from peer 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:37.407222 30055 ts_tablet_manager.cc:902] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: Initiating tablet copy from peer 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:37.407361 30052 tablet_copy_client.cc:308] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.2:32945
I20240501 19:57:37.407768 29399 tablet_copy_service.cc:140] P 26f1ddf6a31645c0a82e697acc9253fb: Received BeginTabletCopySession request for tablet 3e50e51f9e3c4d3d83ae49f7a326551a from peer cbbe61a596804ca093383d53b3a4b2ab ({username='slave'} at 127.0.0.1:40720)
I20240501 19:57:37.407800 30055 tablet_copy_client.cc:308] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.3:44839
I20240501 19:57:37.407887 29399 tablet_copy_service.cc:161] P 26f1ddf6a31645c0a82e697acc9253fb: Beginning new tablet copy session on tablet 3e50e51f9e3c4d3d83ae49f7a326551a from peer cbbe61a596804ca093383d53b3a4b2ab at {username='slave'} at 127.0.0.1:40720: session id = cbbe61a596804ca093383d53b3a4b2ab-3e50e51f9e3c4d3d83ae49f7a326551a
I20240501 19:57:37.408188 29474 tablet_copy_service.cc:140] P 77ff839c08d0479993ba43e53980437b: Received BeginTabletCopySession request for tablet c0d7f921a55e437eaaddf7d276c8de0c from peer ba2031c7573846baa1c0190d340bb485 ({username='slave'} at 127.0.0.1:42058)
I20240501 19:57:37.408265 29474 tablet_copy_service.cc:161] P 77ff839c08d0479993ba43e53980437b: Beginning new tablet copy session on tablet c0d7f921a55e437eaaddf7d276c8de0c from peer ba2031c7573846baa1c0190d340bb485 at {username='slave'} at 127.0.0.1:42058: session id = ba2031c7573846baa1c0190d340bb485-c0d7f921a55e437eaaddf7d276c8de0c
I20240501 19:57:37.408906 29399 tablet_copy_source_session.cc:205] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:37.409085 29474 tablet_copy_source_session.cc:205] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:37.409526 30062 ts_tablet_manager.cc:902] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: Initiating tablet copy from peer 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115)
I20240501 19:57:37.409816 30055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0d7f921a55e437eaaddf7d276c8de0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.409816 30052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e50e51f9e3c4d3d83ae49f7a326551a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.409986 30062 tablet_copy_client.cc:308] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: tablet copy: Beginning tablet copy session from remote peer at address 127.25.83.1:46115
I20240501 19:57:37.411700 30055 tablet_copy_client.cc:783] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:37.411891 30055 tablet_copy_client.cc:647] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:37.412024 30052 tablet_copy_client.cc:783] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:37.412178 30052 tablet_copy_client.cc:647] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:37.412444 29323 tablet_copy_service.cc:140] P 649b8f2f20324033bc021e229cff5774: Received BeginTabletCopySession request for tablet e0518736faac4d54bdb78737484c860a from peer b83077a363e241c8b97248c2617f08dc ({username='slave'} at 127.0.0.1:40612)
I20240501 19:57:37.412528 29323 tablet_copy_service.cc:161] P 649b8f2f20324033bc021e229cff5774: Beginning new tablet copy session on tablet e0518736faac4d54bdb78737484c860a from peer b83077a363e241c8b97248c2617f08dc at {username='slave'} at 127.0.0.1:40612: session id = b83077a363e241c8b97248c2617f08dc-e0518736faac4d54bdb78737484c860a
I20240501 19:57:37.413416 29323 tablet_copy_source_session.cc:205] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:57:37.413479 30056 tablet_copy_client.cc:1002] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:37.413941 30059 tablet_copy_client.cc:1002] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:37.414093 30062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0518736faac4d54bdb78737484c860a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:57:37.415863 30062 tablet_copy_client.cc:783] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: tablet copy: Starting download of 0 data blocks...
I20240501 19:57:37.416002 30062 tablet_copy_client.cc:647] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: tablet copy: Starting download of 1 WAL segments...
I20240501 19:57:37.416939 30065 tablet_copy_client.cc:1002] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: tablet copy: Injecting latency into file download: 2000
I20240501 19:57:37.561236 29264 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:37.572016 29388 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2803395236b144059af4f49d49e8c7e3"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
mode: GRACEFUL
new_leader_uuid: "cbbe61a596804ca093383d53b3a4b2ab"
from {username='slave'} at 127.0.0.1:40732
I20240501 19:57:37.572165 29388 raft_consensus.cc:595] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Received request to transfer leadership to TS cbbe61a596804ca093383d53b3a4b2ab
I20240501 19:57:37.572844 29388 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
mode: GRACEFUL
new_leader_uuid: "649b8f2f20324033bc021e229cff5774"
from {username='slave'} at 127.0.0.1:40732
I20240501 19:57:37.572949 29388 raft_consensus.cc:595] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Received request to transfer leadership to TS 649b8f2f20324033bc021e229cff5774
I20240501 19:57:37.578922 29310 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
new_leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:40616
I20240501 19:57:37.579221 29310 raft_consensus.cc:595] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Received request to transfer leadership to TS 26f1ddf6a31645c0a82e697acc9253fb
I20240501 19:57:37.581672 29310 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
new_leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:40616
I20240501 19:57:37.581941 29310 raft_consensus.cc:595] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Received request to transfer leadership to TS 26f1ddf6a31645c0a82e697acc9253fb
I20240501 19:57:37.582592 29458 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9967547056d848d1b37aff3f6b4abe86"
dest_uuid: "77ff839c08d0479993ba43e53980437b"
mode: GRACEFUL
new_leader_uuid: "ba2031c7573846baa1c0190d340bb485"
from {username='slave'} at 127.0.0.1:42050
I20240501 19:57:37.582706 29458 raft_consensus.cc:595] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Received request to transfer leadership to TS ba2031c7573846baa1c0190d340bb485
I20240501 19:57:37.583318 29458 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd"
dest_uuid: "77ff839c08d0479993ba43e53980437b"
mode: GRACEFUL
new_leader_uuid: "649b8f2f20324033bc021e229cff5774"
from {username='slave'} at 127.0.0.1:42050
I20240501 19:57:37.583415 29458 raft_consensus.cc:595] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Received request to transfer leadership to TS 649b8f2f20324033bc021e229cff5774
I20240501 19:57:37.583662 29264 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 6
I20240501 19:57:37.583792 29264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:37.680888 30030 raft_consensus.cc:979] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb: : Instructing follower cbbe61a596804ca093383d53b3a4b2ab to start an election
I20240501 19:57:37.681022 30030 raft_consensus.cc:1067] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Signalling peer cbbe61a596804ca093383d53b3a4b2ab to start an election
I20240501 19:57:37.681365 29858 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2803395236b144059af4f49d49e8c7e3"
dest_uuid: "cbbe61a596804ca093383d53b3a4b2ab"
from {username='slave'} at 127.0.0.1:40464
I20240501 19:57:37.681509 29858 raft_consensus.cc:482] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:37.681576 29858 raft_consensus.cc:3037] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.682469 29858 raft_consensus.cc:504] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:37.682917 29858 leader_election.cc:288] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [CANDIDATE]: Term 2 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:37.683167 29388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2803395236b144059af4f49d49e8c7e3" candidate_uuid: "cbbe61a596804ca093383d53b3a4b2ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:37.683218 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2803395236b144059af4f49d49e8c7e3" candidate_uuid: "cbbe61a596804ca093383d53b3a4b2ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:37.683265 29388 raft_consensus.cc:3032] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:37.683303 29310 raft_consensus.cc:3037] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.683317 29388 raft_consensus.cc:729] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:37.683455 29388 consensus_queue.cc:260] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:37.683586 29388 raft_consensus.cc:3037] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.684114 29310 raft_consensus.cc:2454] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbbe61a596804ca093383d53b3a4b2ab in term 2.
I20240501 19:57:37.684352 29388 raft_consensus.cc:2454] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbbe61a596804ca093383d53b3a4b2ab in term 2.
I20240501 19:57:37.684654 29821 leader_election.cc:302] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, cbbe61a596804ca093383d53b3a4b2ab; no voters:
I20240501 19:57:37.685010 30072 raft_consensus.cc:2781] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:37.685199 30072 raft_consensus.cc:686] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 2 LEADER]: Becoming Leader. State: Replica: cbbe61a596804ca093383d53b3a4b2ab, State: Running, Role: LEADER
I20240501 19:57:37.685361 30072 consensus_queue.cc:237] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:37.686751 29207 catalog_manager.cc:5484] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab reported cstate change: term changed from 1 to 2, leader changed from 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) to cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6). New cstate: current_term: 2 leader_uuid: "cbbe61a596804ca093383d53b3a4b2ab" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:37.800995 30008 raft_consensus.cc:979] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774: : Instructing follower 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:37.801117 30008 raft_consensus.cc:1067] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Signalling peer 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:37.801438 29388 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:40672
I20240501 19:57:37.801587 29388 raft_consensus.cc:482] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:37.801651 29388 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.802839 29388 raft_consensus.cc:504] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:37.803228 29388 leader_election.cc:288] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 2 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237)
I20240501 19:57:37.803484 29858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "cbbe61a596804ca093383d53b3a4b2ab"
I20240501 19:57:37.803568 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:37.803620 29858 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.803658 29310 raft_consensus.cc:3032] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:37.803701 29310 raft_consensus.cc:729] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:37.803885 29310 consensus_queue.cc:260] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:37.804041 29310 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.804589 29858 raft_consensus.cc:2454] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 2.
I20240501 19:57:37.804870 29355 leader_election.cc:302] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, cbbe61a596804ca093383d53b3a4b2ab; no voters:
I20240501 19:57:37.805030 29310 raft_consensus.cc:2454] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 2.
I20240501 19:57:37.805109 30030 raft_consensus.cc:2781] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:37.805198 30030 raft_consensus.cc:686] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Becoming Leader. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:37.805313 30030 consensus_queue.cc:237] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:37.806794 29207 catalog_manager.cc:5484] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: term changed from 1 to 2, leader changed from 649b8f2f20324033bc021e229cff5774 (127.25.83.1) to 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2). New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:37.825767 30069 raft_consensus.cc:979] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b: : Instructing follower 649b8f2f20324033bc021e229cff5774 to start an election
I20240501 19:57:37.825866 30069 raft_consensus.cc:1067] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Signalling peer 649b8f2f20324033bc021e229cff5774 to start an election
I20240501 19:57:37.826251 29310 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
from {username='slave'} at 127.0.0.1:40564
I20240501 19:57:37.826364 29310 raft_consensus.cc:482] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:37.826421 29310 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.827312 29310 raft_consensus.cc:504] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:37.827658 29310 leader_election.cc:288] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 2 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:37.827931 29458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:37.827994 29387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:37.828064 29387 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.828064 29458 raft_consensus.cc:3032] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:37.828140 29458 raft_consensus.cc:729] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:37.828280 29458 consensus_queue.cc:260] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:37.828464 29458 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.828997 29387 raft_consensus.cc:2454] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 2.
I20240501 19:57:37.829310 29276 leader_election.cc:302] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:37.829378 29458 raft_consensus.cc:2454] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 2.
I20240501 19:57:37.829486 30081 raft_consensus.cc:2781] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:37.829568 30081 raft_consensus.cc:686] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Becoming Leader. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:37.829912 30081 consensus_queue.cc:237] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:37.831815 29207 catalog_manager.cc:5484] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 reported cstate change: term changed from 1 to 2, leader changed from 77ff839c08d0479993ba43e53980437b (127.25.83.3) to 649b8f2f20324033bc021e229cff5774 (127.25.83.1). New cstate: current_term: 2 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:37.835739 29310 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:37.835850 29310 raft_consensus.cc:595] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Received request to transfer leadership
I20240501 19:57:37.839885 29310 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:37.839983 29310 raft_consensus.cc:595] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Received request to transfer leadership
W20240501 19:57:37.840247 29195 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for a2f89b5db8c34edaa8751a263dc818cd already in progress
I20240501 19:57:37.840869 29310 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:37.840962 29310 raft_consensus.cc:595] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Received request to transfer leadership
W20240501 19:57:37.841256 29258 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for a2f89b5db8c34edaa8751a263dc818cd already in progress
W20240501 19:57:37.841310 29258 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for a2f89b5db8c34edaa8751a263dc818cd already in progress
I20240501 19:57:37.875433 30069 raft_consensus.cc:979] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b: : Instructing follower ba2031c7573846baa1c0190d340bb485 to start an election
I20240501 19:57:37.875566 30069 raft_consensus.cc:1067] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Signalling peer ba2031c7573846baa1c0190d340bb485 to start an election
I20240501 19:57:37.875928 29783 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9967547056d848d1b37aff3f6b4abe86"
dest_uuid: "ba2031c7573846baa1c0190d340bb485"
from {username='slave'} at 127.0.0.1:50000
I20240501 19:57:37.876080 29783 raft_consensus.cc:482] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:37.876153 29783 raft_consensus.cc:3037] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.877182 29783 raft_consensus.cc:504] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:37.877612 29783 leader_election.cc:288] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [CANDIDATE]: Term 2 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:37.877844 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9967547056d848d1b37aff3f6b4abe86" candidate_uuid: "ba2031c7573846baa1c0190d340bb485" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:37.877947 29310 raft_consensus.cc:3037] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.878082 29458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9967547056d848d1b37aff3f6b4abe86" candidate_uuid: "ba2031c7573846baa1c0190d340bb485" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:37.878276 29458 raft_consensus.cc:3032] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:37.878358 29458 raft_consensus.cc:729] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:37.878542 29458 consensus_queue.cc:260] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:37.878767 29458 raft_consensus.cc:3037] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.878911 29310 raft_consensus.cc:2454] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba2031c7573846baa1c0190d340bb485 in term 2.
I20240501 19:57:37.879189 29747 leader_election.cc:302] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, ba2031c7573846baa1c0190d340bb485; no voters:
I20240501 19:57:37.879520 30084 raft_consensus.cc:2781] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:37.879789 29458 raft_consensus.cc:2454] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba2031c7573846baa1c0190d340bb485 in term 2.
I20240501 19:57:37.879984 30084 raft_consensus.cc:686] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 2 LEADER]: Becoming Leader. State: Replica: ba2031c7573846baa1c0190d340bb485, State: Running, Role: LEADER
I20240501 19:57:37.880162 30084 consensus_queue.cc:237] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:37.881644 29207 catalog_manager.cc:5484] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 reported cstate change: term changed from 1 to 2, leader changed from 77ff839c08d0479993ba43e53980437b (127.25.83.3) to ba2031c7573846baa1c0190d340bb485 (127.25.83.5). New cstate: current_term: 2 leader_uuid: "ba2031c7573846baa1c0190d340bb485" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:37.917834 30081 raft_consensus.cc:979] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774: : Instructing follower 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:37.917968 30081 raft_consensus.cc:1067] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Signalling peer 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:37.918376 29387 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:40672
I20240501 19:57:37.918515 29387 raft_consensus.cc:482] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:37.918582 29387 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.919763 29387 raft_consensus.cc:504] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:37.920184 29387 leader_election.cc:288] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 2 election: Requested vote from peers 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839), 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115)
I20240501 19:57:37.920353 29458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:37.920550 29458 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.920676 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:37.920780 29310 raft_consensus.cc:3032] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:37.921008 29310 raft_consensus.cc:729] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:37.921154 29310 consensus_queue.cc:260] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:37.921397 29310 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.921502 29458 raft_consensus.cc:2454] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 2.
I20240501 19:57:37.921751 29354 leader_election.cc:302] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:37.921965 30087 raft_consensus.cc:2781] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:37.922049 30087 raft_consensus.cc:686] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Becoming Leader. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:37.922201 30087 consensus_queue.cc:237] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:37.922350 29310 raft_consensus.cc:2454] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 2.
I20240501 19:57:37.923959 29207 catalog_manager.cc:5484] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: term changed from 1 to 2, leader changed from 649b8f2f20324033bc021e229cff5774 (127.25.83.1) to 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2). New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:37.927703 30030 raft_consensus.cc:979] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb: : Instructing follower 649b8f2f20324033bc021e229cff5774 to start an election
I20240501 19:57:37.927793 30030 raft_consensus.cc:1067] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Signalling peer 649b8f2f20324033bc021e229cff5774 to start an election
I20240501 19:57:37.928117 29309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
from {username='slave'} at 127.0.0.1:40558
I20240501 19:57:37.928205 29309 raft_consensus.cc:482] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:37.928256 29309 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.929044 29309 raft_consensus.cc:504] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:37.929363 29309 leader_election.cc:288] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 2 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:37.929651 29388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:37.929679 29458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:37.929737 29388 raft_consensus.cc:3032] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:37.929934 29458 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.929948 29388 raft_consensus.cc:729] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:37.930168 29388 consensus_queue.cc:260] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:37.930357 29388 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:37.930807 29458 raft_consensus.cc:2454] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 2.
I20240501 19:57:37.931044 29278 leader_election.cc:302] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:37.931162 29388 raft_consensus.cc:2454] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 2.
I20240501 19:57:37.931229 30082 raft_consensus.cc:2781] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:37.931344 30082 raft_consensus.cc:686] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Becoming Leader. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:37.931517 30082 consensus_queue.cc:237] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:37.933133 29207 catalog_manager.cc:5484] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 reported cstate change: term changed from 1 to 2, leader changed from 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) to 649b8f2f20324033bc021e229cff5774 (127.25.83.1). New cstate: current_term: 2 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:37.935679 29309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:37.935784 29309 raft_consensus.cc:595] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Received request to transfer leadership
I20240501 19:57:37.939769 29309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:37.939882 29309 raft_consensus.cc:595] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Received request to transfer leadership
I20240501 19:57:37.940518 29309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:37.940629 29309 raft_consensus.cc:595] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Received request to transfer leadership
W20240501 19:57:37.940871 29258 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 3e50e51f9e3c4d3d83ae49f7a326551a already in progress
W20240501 19:57:37.940917 29258 auto_rebalancer.cc:263] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 3e50e51f9e3c4d3d83ae49f7a326551a already in progress
I20240501 19:57:38.127931 29388 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Refusing update from remote peer cbbe61a596804ca093383d53b3a4b2ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:38.128306 30072 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:38.131487 29309 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer cbbe61a596804ca093383d53b3a4b2ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:38.132019 30072 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:38.227505 29858 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:38.227887 30087 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:38.230518 29310 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:38.230844 30030 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240501 19:57:38.284879 29279 consensus_peers.cc:485] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:38.324270 29464 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 19:57:38.324663 30087 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240501 19:57:38.326927 29353 consensus_peers.cc:485] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb -> Peer b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171): Couldn't send request to peer b83077a363e241c8b97248c2617f08dc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:38.327383 29310 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20240501 19:57:38.327798 30087 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20240501 19:57:38.330158 29279 consensus_peers.cc:485] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:38.337595 29464 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer ba2031c7573846baa1c0190d340bb485: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:38.337991 30084 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:38.340390 29310 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer ba2031c7573846baa1c0190d340bb485: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:38.340814 30102 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:38.363047 29464 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 19:57:38.363480 30081 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:38.365973 30082 raft_consensus.cc:979] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: : Instructing follower 77ff839c08d0479993ba43e53980437b to start an election
I20240501 19:57:38.366070 30082 raft_consensus.cc:1067] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Signalling peer 77ff839c08d0479993ba43e53980437b to start an election
I20240501 19:57:38.366564 29464 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a"
dest_uuid: "77ff839c08d0479993ba43e53980437b"
from {username='slave'} at 127.0.0.1:42018
I20240501 19:57:38.366681 29464 raft_consensus.cc:482] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:38.366734 29464 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:38.367156 29388 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 19:57:38.367439 30081 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:38.367825 29464 raft_consensus.cc:504] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:38.368202 29464 leader_election.cc:288] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 3 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
I20240501 19:57:38.368537 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:38.368640 29387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e50e51f9e3c4d3d83ae49f7a326551a" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:38.368639 29309 raft_consensus.cc:3032] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:57:38.368911 29309 raft_consensus.cc:729] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:38.369048 29309 consensus_queue.cc:260] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:38.369180 29309 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:38.370616 29387 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 19:57:38.370999 30081 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:38.371065 29309 raft_consensus.cc:2454] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 3.
I20240501 19:57:38.371292 29428 leader_election.cc:302] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, 77ff839c08d0479993ba43e53980437b; no voters: 26f1ddf6a31645c0a82e697acc9253fb
I20240501 19:57:38.371505 30069 raft_consensus.cc:2781] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:38.371588 30069 raft_consensus.cc:686] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 3 LEADER]: Becoming Leader. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:38.371708 30069 consensus_queue.cc:237] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:38.372822 30082 raft_consensus.cc:979] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: : Instructing follower 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:38.372895 30082 raft_consensus.cc:1067] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Signalling peer 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:38.373716 29210 catalog_manager.cc:5484] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b reported cstate change: term changed from 2 to 3, leader changed from 649b8f2f20324033bc021e229cff5774 (127.25.83.1) to 77ff839c08d0479993ba43e53980437b (127.25.83.3). New cstate: current_term: 3 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:38.373886 29458 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 19:57:38.374284 30080 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:38.374877 29388 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:40672
I20240501 19:57:38.374985 29388 raft_consensus.cc:482] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:38.375041 29388 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:38.375916 29388 raft_consensus.cc:504] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:38.376183 29388 leader_election.cc:288] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 3 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:38.376430 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:38.376439 29458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f89b5db8c34edaa8751a263dc818cd" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:38.376571 29458 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:38.376571 29309 raft_consensus.cc:3032] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:57:38.376645 29309 raft_consensus.cc:729] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:38.376773 29309 consensus_queue.cc:260] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:38.376932 29309 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:38.377526 29458 raft_consensus.cc:2454] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 3.
I20240501 19:57:38.377802 29309 raft_consensus.cc:2454] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 3.
I20240501 19:57:38.377811 29354 leader_election.cc:302] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:38.378312 30087 raft_consensus.cc:2781] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:38.378397 30087 raft_consensus.cc:686] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 3 LEADER]: Becoming Leader. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:38.378528 30087 consensus_queue.cc:237] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:38.380034 29210 catalog_manager.cc:5484] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: term changed from 2 to 3, leader changed from 649b8f2f20324033bc021e229cff5774 (127.25.83.1) to 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2). New cstate: current_term: 3 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:38.781800 29309 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 19:57:38.782217 30069 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:38.784226 29387 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Advancing to term 3
W20240501 19:57:38.784930 29430 consensus_peers.cc:485] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:38.785338 29387 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 19:57:38.786314 30070 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:38.874053 29458 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 19:57:38.874502 30087 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240501 19:57:38.876788 29355 consensus_peers.cc:485] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:38.877292 29310 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20240501 19:57:38.877703 30071 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:39.223258 30033 tablet_copy_client.cc:523] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:39.224525 30033 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: Bootstrap starting.
I20240501 19:57:39.246676 30033 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:39.246985 30033 tablet_bootstrap.cc:492] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: Bootstrap complete.
I20240501 19:57:39.247189 30033 ts_tablet_manager.cc:1367] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: Time spent bootstrapping tablet: real 0.023s user 0.023s sys 0.000s
I20240501 19:57:39.247659 30033 raft_consensus.cc:348] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:39.247812 30033 raft_consensus.cc:729] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba2031c7573846baa1c0190d340bb485, State: Initialized, Role: LEARNER
I20240501 19:57:39.247910 30033 consensus_queue.cc:260] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:39.248172 30033 ts_tablet_manager.cc:1398] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:39.248571 29323 tablet_copy_service.cc:342] P 649b8f2f20324033bc021e229cff5774: Request end of tablet copy session ba2031c7573846baa1c0190d340bb485-39fb3d9c368d4c0d95bf935f60f863e8 received from {username='slave'} at 127.0.0.1:40596
I20240501 19:57:39.248647 29323 tablet_copy_service.cc:433] P 649b8f2f20324033bc021e229cff5774: ending tablet copy session ba2031c7573846baa1c0190d340bb485-39fb3d9c368d4c0d95bf935f60f863e8 on tablet 39fb3d9c368d4c0d95bf935f60f863e8 with peer ba2031c7573846baa1c0190d340bb485
I20240501 19:57:39.282842 30038 tablet_copy_client.cc:523] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:39.284549 30038 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: Bootstrap starting.
I20240501 19:57:39.286146 30045 tablet_copy_client.cc:523] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:39.287122 30045 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap starting.
I20240501 19:57:39.306947 30038 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=139 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:39.307238 30038 tablet_bootstrap.cc:492] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: Bootstrap complete.
I20240501 19:57:39.307471 30038 ts_tablet_manager.cc:1367] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.005s
I20240501 19:57:39.307896 30038 raft_consensus.cc:348] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:39.308099 30038 raft_consensus.cc:729] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b83077a363e241c8b97248c2617f08dc, State: Initialized, Role: LEARNER
I20240501 19:57:39.308264 30038 consensus_queue.cc:260] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:39.308562 30038 ts_tablet_manager.cc:1398] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:57:39.309034 29323 tablet_copy_service.cc:342] P 649b8f2f20324033bc021e229cff5774: Request end of tablet copy session b83077a363e241c8b97248c2617f08dc-5fe56fe5bfe2443c97aa252754a24aa6 received from {username='slave'} at 127.0.0.1:40612
I20240501 19:57:39.309109 29323 tablet_copy_service.cc:433] P 649b8f2f20324033bc021e229cff5774: ending tablet copy session b83077a363e241c8b97248c2617f08dc-5fe56fe5bfe2443c97aa252754a24aa6 on tablet 5fe56fe5bfe2443c97aa252754a24aa6 with peer b83077a363e241c8b97248c2617f08dc
I20240501 19:57:39.314342 30045 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=98 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:39.314590 30045 tablet_bootstrap.cc:492] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap complete.
I20240501 19:57:39.314767 30045 ts_tablet_manager.cc:1367] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.009s
I20240501 19:57:39.315136 30045 raft_consensus.cc:348] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:39.315287 30045 raft_consensus.cc:729] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbbe61a596804ca093383d53b3a4b2ab, State: Initialized, Role: LEARNER
I20240501 19:57:39.315399 30045 consensus_queue.cc:260] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:39.315701 30045 ts_tablet_manager.cc:1398] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:57:39.316130 29474 tablet_copy_service.cc:342] P 77ff839c08d0479993ba43e53980437b: Request end of tablet copy session cbbe61a596804ca093383d53b3a4b2ab-a2f89b5db8c34edaa8751a263dc818cd received from {username='slave'} at 127.0.0.1:42064
I20240501 19:57:39.316205 29474 tablet_copy_service.cc:433] P 77ff839c08d0479993ba43e53980437b: ending tablet copy session cbbe61a596804ca093383d53b3a4b2ab-a2f89b5db8c34edaa8751a263dc818cd on tablet a2f89b5db8c34edaa8751a263dc818cd with peer cbbe61a596804ca093383d53b3a4b2ab
I20240501 19:57:39.336182 29783 raft_consensus.cc:1203] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.28->[1.29-1.29] Dedup: 1.29->[]
I20240501 19:57:39.353356 29707 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Advancing to term 2
I20240501 19:57:39.413756 30052 tablet_copy_client.cc:523] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:39.414207 30055 tablet_copy_client.cc:523] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:39.415445 30052 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap starting.
I20240501 19:57:39.415534 30055 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: Bootstrap starting.
I20240501 19:57:39.416591 29858 raft_consensus.cc:3037] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Advancing to term 3
I20240501 19:57:39.417196 30062 tablet_copy_client.cc:523] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:57:39.417526 29858 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [term 3 LEARNER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 19:57:39.417915 30087 consensus_queue.cc:1037] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:39.418489 30062 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: Bootstrap starting.
I20240501 19:57:39.434883 30087 raft_consensus.cc:1050] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb: attempting to promote NON_VOTER cbbe61a596804ca093383d53b3a4b2ab to VOTER
I20240501 19:57:39.435281 30087 consensus_queue.cc:237] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:39.436822 29309 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 19:57:39.437034 29464 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 19:57:39.437167 30030 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:39.437503 29858 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [term 3 LEARNER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20240501 19:57:39.438098 30030 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:39.438369 30030 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:39.439842 30030 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 27 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.441252 30052 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=107 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:39.441716 29858 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.442013 29310 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.442196 30055 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=94 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:39.442515 30055 tablet_bootstrap.cc:492] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: Bootstrap complete.
I20240501 19:57:39.442790 30055 ts_tablet_manager.cc:1367] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20240501 19:57:39.445164 30055 raft_consensus.cc:348] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:39.445281 30055 raft_consensus.cc:729] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ba2031c7573846baa1c0190d340bb485, State: Initialized, Role: LEARNER
I20240501 19:57:39.445202 29207 catalog_manager.cc:5484] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index 27 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:39.445379 30055 consensus_queue.cc:260] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:39.445675 30055 ts_tablet_manager.cc:1398] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:57:39.445858 30052 tablet_bootstrap.cc:492] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: Bootstrap complete.
I20240501 19:57:39.446339 30052 ts_tablet_manager.cc:1367] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: Time spent bootstrapping tablet: real 0.031s user 0.021s sys 0.002s
I20240501 19:57:39.446619 29474 tablet_copy_service.cc:342] P 77ff839c08d0479993ba43e53980437b: Request end of tablet copy session ba2031c7573846baa1c0190d340bb485-c0d7f921a55e437eaaddf7d276c8de0c received from {username='slave'} at 127.0.0.1:42058
I20240501 19:57:39.446842 29474 tablet_copy_service.cc:433] P 77ff839c08d0479993ba43e53980437b: ending tablet copy session ba2031c7573846baa1c0190d340bb485-c0d7f921a55e437eaaddf7d276c8de0c on tablet c0d7f921a55e437eaaddf7d276c8de0c with peer ba2031c7573846baa1c0190d340bb485
I20240501 19:57:39.447458 30052 raft_consensus.cc:348] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:39.447692 30052 raft_consensus.cc:729] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cbbe61a596804ca093383d53b3a4b2ab, State: Initialized, Role: LEARNER
I20240501 19:57:39.447886 30052 consensus_queue.cc:260] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:39.448441 30052 ts_tablet_manager.cc:1398] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 19:57:39.449368 29399 tablet_copy_service.cc:342] P 26f1ddf6a31645c0a82e697acc9253fb: Request end of tablet copy session cbbe61a596804ca093383d53b3a4b2ab-3e50e51f9e3c4d3d83ae49f7a326551a received from {username='slave'} at 127.0.0.1:40720
I20240501 19:57:39.449440 29399 tablet_copy_service.cc:433] P 26f1ddf6a31645c0a82e697acc9253fb: ending tablet copy session cbbe61a596804ca093383d53b3a4b2ab-3e50e51f9e3c4d3d83ae49f7a326551a on tablet 3e50e51f9e3c4d3d83ae49f7a326551a with peer cbbe61a596804ca093383d53b3a4b2ab
I20240501 19:57:39.449858 29458 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.453673 29387 consensus_queue.cc:237] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:39.454732 29858 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [term 3 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 19:57:39.454779 29458 raft_consensus.cc:1261] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 19:57:39.455224 30087 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.455607 30062 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:57:39.455847 30062 tablet_bootstrap.cc:492] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: Bootstrap complete.
I20240501 19:57:39.456013 30062 ts_tablet_manager.cc:1367] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: Time spent bootstrapping tablet: real 0.038s user 0.025s sys 0.003s
I20240501 19:57:39.456437 30062 raft_consensus.cc:348] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:39.456555 30062 raft_consensus.cc:729] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b83077a363e241c8b97248c2617f08dc, State: Initialized, Role: LEARNER
I20240501 19:57:39.456658 30062 consensus_queue.cc:260] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:39.456939 30062 ts_tablet_manager.cc:1398] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:57:39.457345 29323 tablet_copy_service.cc:342] P 649b8f2f20324033bc021e229cff5774: Request end of tablet copy session b83077a363e241c8b97248c2617f08dc-e0518736faac4d54bdb78737484c860a received from {username='slave'} at 127.0.0.1:40612
I20240501 19:57:39.457427 29323 tablet_copy_service.cc:433] P 649b8f2f20324033bc021e229cff5774: ending tablet copy session b83077a363e241c8b97248c2617f08dc-e0518736faac4d54bdb78737484c860a on tablet e0518736faac4d54bdb78737484c860a with peer b83077a363e241c8b97248c2617f08dc
I20240501 19:57:39.457619 30125 consensus_queue.cc:1035] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.458176 30125 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.459452 29194 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2f89b5db8c34edaa8751a263dc818cd with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:39.460098 29464 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.461499 29858 raft_consensus.cc:2932] T a2f89b5db8c34edaa8751a263dc818cd P cbbe61a596804ca093383d53b3a4b2ab [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.463037 29205 catalog_manager.cc:5484] T a2f89b5db8c34edaa8751a263dc818cd P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index 30 to 31, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New cstate: current_term: 3 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:39.465534 29300 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f89b5db8c34edaa8751a263dc818cd with delete_type TABLET_DATA_TOMBSTONED (TS 649b8f2f20324033bc021e229cff5774 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:39.465873 30130 tablet_replica.cc:330] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: stopping tablet replica
I20240501 19:57:39.466010 30130 raft_consensus.cc:2229] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:39.466233 30130 raft_consensus.cc:2258] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:39.466912 30130 ts_tablet_manager.cc:1875] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:39.468878 30130 ts_tablet_manager.cc:1888] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20240501 19:57:39.468957 30130 log.cc:1196] T a2f89b5db8c34edaa8751a263dc818cd P 649b8f2f20324033bc021e229cff5774: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root/wals/a2f89b5db8c34edaa8751a263dc818cd
I20240501 19:57:39.469537 29195 catalog_manager.cc:4830] TS 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115): tablet a2f89b5db8c34edaa8751a263dc818cd (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:39.477430 29783 raft_consensus.cc:1203] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20240501 19:57:39.584060 29264 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:39.586400 29387 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
mode: GRACEFUL
new_leader_uuid: "649b8f2f20324033bc021e229cff5774"
from {username='slave'} at 127.0.0.1:40732
I20240501 19:57:39.586585 29387 raft_consensus.cc:595] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Received request to transfer leadership to TS 649b8f2f20324033bc021e229cff5774
I20240501 19:57:39.587320 29464 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8cd1acf0afe14cc2a388fcf79ddae20d"
dest_uuid: "77ff839c08d0479993ba43e53980437b"
mode: GRACEFUL
new_leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:42050
I20240501 19:57:39.587436 29464 raft_consensus.cc:595] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Received request to transfer leadership to TS 26f1ddf6a31645c0a82e697acc9253fb
I20240501 19:57:39.588094 29387 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
mode: GRACEFUL
new_leader_uuid: "b83077a363e241c8b97248c2617f08dc"
from {username='slave'} at 127.0.0.1:40732
I20240501 19:57:39.588229 29387 raft_consensus.cc:595] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Received request to transfer leadership to TS b83077a363e241c8b97248c2617f08dc
I20240501 19:57:39.588937 29387 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
mode: GRACEFUL
new_leader_uuid: "77ff839c08d0479993ba43e53980437b"
from {username='slave'} at 127.0.0.1:40732
I20240501 19:57:39.589043 29387 raft_consensus.cc:595] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Received request to transfer leadership to TS 77ff839c08d0479993ba43e53980437b
I20240501 19:57:39.589376 29264 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 4
I20240501 19:57:39.589526 29264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:39.671504 30070 raft_consensus.cc:979] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b: : Instructing follower 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:39.671648 30070 raft_consensus.cc:1067] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Signalling peer 26f1ddf6a31645c0a82e697acc9253fb to start an election
I20240501 19:57:39.672057 29387 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8cd1acf0afe14cc2a388fcf79ddae20d"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:40682
I20240501 19:57:39.672188 29387 raft_consensus.cc:482] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:39.672250 29387 raft_consensus.cc:3037] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:39.673312 29387 raft_consensus.cc:504] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:39.673673 29387 leader_election.cc:288] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 2 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:39.673924 29310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd1acf0afe14cc2a388fcf79ddae20d" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:39.673954 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cd1acf0afe14cc2a388fcf79ddae20d" candidate_uuid: "26f1ddf6a31645c0a82e697acc9253fb" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:39.674032 29310 raft_consensus.cc:3037] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:39.674085 29464 raft_consensus.cc:3032] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:39.674154 29464 raft_consensus.cc:729] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:39.674299 29464 consensus_queue.cc:260] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:39.674437 29464 raft_consensus.cc:3037] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:39.674803 29310 raft_consensus.cc:2454] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 2.
I20240501 19:57:39.675081 29353 leader_election.cc:302] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 649b8f2f20324033bc021e229cff5774; no voters:
I20240501 19:57:39.675268 30071 raft_consensus.cc:2781] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:39.675359 30071 raft_consensus.cc:686] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Becoming Leader. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:39.675479 30071 consensus_queue.cc:237] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } }
I20240501 19:57:39.676196 29464 raft_consensus.cc:2454] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f1ddf6a31645c0a82e697acc9253fb in term 2.
I20240501 19:57:39.677065 29202 catalog_manager.cc:5484] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: term changed from 1 to 2, leader changed from 77ff839c08d0479993ba43e53980437b (127.25.83.3) to 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2). New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:39.731448 30030 raft_consensus.cc:979] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb: : Instructing follower 649b8f2f20324033bc021e229cff5774 to start an election
I20240501 19:57:39.731607 30030 raft_consensus.cc:1067] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Signalling peer 649b8f2f20324033bc021e229cff5774 to start an election
I20240501 19:57:39.732090 29309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
from {username='slave'} at 127.0.0.1:40558
I20240501 19:57:39.732249 29309 raft_consensus.cc:482] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:39.732324 29309 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:39.734194 29309 raft_consensus.cc:504] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:39.735240 29309 leader_election.cc:288] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 3 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237)
I20240501 19:57:39.735909 29858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "cbbe61a596804ca093383d53b3a4b2ab"
I20240501 19:57:39.736050 29858 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:39.737274 29858 raft_consensus.cc:2454] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 3.
I20240501 19:57:39.737530 29387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f49bfbb5624aa89fcdfb1e8eaecc87" candidate_uuid: "649b8f2f20324033bc021e229cff5774" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: true dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:39.737645 29387 raft_consensus.cc:3032] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:57:39.737692 29387 raft_consensus.cc:729] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:39.737825 29279 leader_election.cc:302] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, cbbe61a596804ca093383d53b3a4b2ab; no voters:
I20240501 19:57:39.737859 29387 consensus_queue.cc:260] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:39.738052 30080 raft_consensus.cc:2781] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:39.738066 29387 raft_consensus.cc:3037] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:39.738283 30080 raft_consensus.cc:686] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [term 3 LEADER]: Becoming Leader. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:39.738497 30080 consensus_queue.cc:237] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:39.739333 29387 raft_consensus.cc:2454] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649b8f2f20324033bc021e229cff5774 in term 3.
I20240501 19:57:39.740080 29202 catalog_manager.cc:5484] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 reported cstate change: term changed from 2 to 3, leader changed from 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) to 649b8f2f20324033bc021e229cff5774 (127.25.83.1). New cstate: current_term: 3 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:39.786394 29858 raft_consensus.cc:3037] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [term 1 LEARNER]: Advancing to term 3
I20240501 19:57:39.787593 29858 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [term 3 LEARNER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 19:57:39.787973 30070 consensus_queue.cc:1037] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:39.789690 30070 raft_consensus.cc:1050] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b: attempting to promote NON_VOTER cbbe61a596804ca093383d53b3a4b2ab to VOTER
I20240501 19:57:39.790040 30070 consensus_queue.cc:237] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:39.791344 29387 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 19:57:39.791409 29309 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 19:57:39.791781 30137 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.792029 30137 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.792093 29858 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [term 3 LEARNER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 19:57:39.792497 30136 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.794245 30114 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.795853 29310 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.796289 29387 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.797873 29202 catalog_manager.cc:5484] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index 28 to 31, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:39.798076 29858 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 28 to 31, cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.800694 29464 consensus_queue.cc:237] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } }
I20240501 19:57:39.801621 29858 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [term 3 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 19:57:39.801821 29387 raft_consensus.cc:1261] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 19:57:39.801963 30137 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:39.802330 30136 consensus_queue.cc:1035] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:39.803903 30114 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P 77ff839c08d0479993ba43e53980437b [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.804364 29858 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.805197 29196 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e50e51f9e3c4d3d83ae49f7a326551a with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:39.805863 29387 raft_consensus.cc:2932] T 3e50e51f9e3c4d3d83ae49f7a326551a P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.806144 29210 catalog_manager.cc:5484] T 3e50e51f9e3c4d3d83ae49f7a326551a P cbbe61a596804ca093383d53b3a4b2ab reported cstate change: config changed from index 31 to 32, VOTER 649b8f2f20324033bc021e229cff5774 (127.25.83.1) evicted. New cstate: current_term: 3 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } }
I20240501 19:57:39.808355 29300 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e50e51f9e3c4d3d83ae49f7a326551a with delete_type TABLET_DATA_TOMBSTONED (TS 649b8f2f20324033bc021e229cff5774 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:40546
I20240501 19:57:39.808574 30130 tablet_replica.cc:330] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: stopping tablet replica
I20240501 19:57:39.808730 30130 raft_consensus.cc:2229] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:39.808880 30130 raft_consensus.cc:2258] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:39.809499 30130 ts_tablet_manager.cc:1875] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:39.811450 30130 ts_tablet_manager.cc:1888] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20240501 19:57:39.811528 30130 log.cc:1196] T 3e50e51f9e3c4d3d83ae49f7a326551a P 649b8f2f20324033bc021e229cff5774: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-0-root/wals/3e50e51f9e3c4d3d83ae49f7a326551a
I20240501 19:57:39.811957 29195 catalog_manager.cc:4830] TS 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115): tablet 3e50e51f9e3c4d3d83ae49f7a326551a (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:39.901252 30132 raft_consensus.cc:1050] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774: attempting to promote NON_VOTER ba2031c7573846baa1c0190d340bb485 to VOTER
I20240501 19:57:39.901609 30132 consensus_queue.cc:237] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:39.902704 29783 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:39.903020 29458 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:39.903020 29387 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:39.902972 30140 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:39.903353 30142 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:39.903709 30140 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:39.905165 30132 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:39.905516 29707 raft_consensus.cc:1203] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.30->[1.31-1.31] Dedup: 1.31->[]
I20240501 19:57:39.906538 29388 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:39.907538 29205 catalog_manager.cc:5484] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:39.909024 29783 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:39.911737 29458 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:39.912817 29309 consensus_queue.cc:237] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:39.913857 29783 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:39.914089 29458 raft_consensus.cc:1261] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20240501 19:57:39.914188 30080 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.914479 30080 consensus_queue.cc:1035] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.916375 30142 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:39.917176 29783 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:39.919148 29464 raft_consensus.cc:2932] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:39.921545 29205 catalog_manager.cc:5484] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 reported cstate change: config changed from index 30 to 31, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:39.922689 29195 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39fb3d9c368d4c0d95bf935f60f863e8 with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:39.923741 29376 tablet_service.cc:1514] Processing DeleteTablet for tablet 39fb3d9c368d4c0d95bf935f60f863e8 with delete_type TABLET_DATA_TOMBSTONED (TS 26f1ddf6a31645c0a82e697acc9253fb not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:40664
I20240501 19:57:39.924113 30146 tablet_replica.cc:330] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: stopping tablet replica
I20240501 19:57:39.924249 30146 raft_consensus.cc:2229] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:39.924436 30146 raft_consensus.cc:2258] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:39.925516 30146 ts_tablet_manager.cc:1875] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:39.927208 30146 ts_tablet_manager.cc:1888] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.30
I20240501 19:57:39.927274 30146 log.cc:1196] T 39fb3d9c368d4c0d95bf935f60f863e8 P 26f1ddf6a31645c0a82e697acc9253fb: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root/wals/39fb3d9c368d4c0d95bf935f60f863e8
I20240501 19:57:39.928023 29194 catalog_manager.cc:4830] TS 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945): tablet 39fb3d9c368d4c0d95bf935f60f863e8 (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:39.933593 30030 raft_consensus.cc:979] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb: : Instructing follower b83077a363e241c8b97248c2617f08dc to start an election
I20240501 19:57:39.933701 30071 raft_consensus.cc:1067] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Signalling peer b83077a363e241c8b97248c2617f08dc to start an election
I20240501 19:57:39.934075 29707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0"
dest_uuid: "b83077a363e241c8b97248c2617f08dc"
from {username='slave'} at 127.0.0.1:48612
I20240501 19:57:39.934222 29707 raft_consensus.cc:482] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:39.934288 29707 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:39.935153 29707 raft_consensus.cc:504] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:39.935607 29707 leader_election.cc:288] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [CANDIDATE]: Term 3 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:39.935890 29388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0" candidate_uuid: "b83077a363e241c8b97248c2617f08dc" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:39.935986 29388 raft_consensus.cc:3032] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:57:39.936025 29388 raft_consensus.cc:729] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:39.936148 29388 consensus_queue.cc:260] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:39.936262 29388 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:39.936866 29388 raft_consensus.cc:2454] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83077a363e241c8b97248c2617f08dc in term 3.
I20240501 19:57:39.937126 29670 leader_election.cc:302] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, b83077a363e241c8b97248c2617f08dc; no voters:
I20240501 19:57:39.937310 30115 raft_consensus.cc:2781] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:39.937449 30115 raft_consensus.cc:686] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 3 LEADER]: Becoming Leader. State: Replica: b83077a363e241c8b97248c2617f08dc, State: Running, Role: LEADER
I20240501 19:57:39.937587 30115 consensus_queue.cc:237] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:39.938961 29210 catalog_manager.cc:5484] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc reported cstate change: term changed from 2 to 3, leader changed from 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) to b83077a363e241c8b97248c2617f08dc (127.25.83.4). New cstate: current_term: 3 leader_uuid: "b83077a363e241c8b97248c2617f08dc" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:39.944648 29464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b886671d00534706ad4fc5f6023c1dd0" candidate_uuid: "b83077a363e241c8b97248c2617f08dc" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:39.944756 29464 raft_consensus.cc:3037] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:39.944834 30030 raft_consensus.cc:1050] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb: attempting to promote NON_VOTER b83077a363e241c8b97248c2617f08dc to VOTER
I20240501 19:57:39.945245 30030 consensus_queue.cc:237] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:39.947365 29464 raft_consensus.cc:2454] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83077a363e241c8b97248c2617f08dc in term 3.
I20240501 19:57:39.949580 29458 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 19:57:39.949597 29309 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 19:57:39.949996 30030 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.950074 29707 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 2 LEARNER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 19:57:39.950322 30071 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.950489 30087 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:39.952191 30030 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:39.952713 29309 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:39.953013 29464 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:39.953967 29707 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:39.954758 29210 catalog_manager.cc:5484] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index 29 to 31, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:39.957444 29388 consensus_queue.cc:237] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:39.959004 29707 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:39.959069 29309 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:39.959381 30030 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:39.959553 30125 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:39.961015 30087 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:39.962575 29194 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5fe56fe5bfe2443c97aa252754a24aa6 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:39.962779 29707 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:39.962966 29309 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:39.963192 29205 catalog_manager.cc:5484] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index 31 to 32, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:39.965481 29454 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fe56fe5bfe2443c97aa252754a24aa6 with delete_type TABLET_DATA_TOMBSTONED (TS 77ff839c08d0479993ba43e53980437b not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:42002
I20240501 19:57:39.965924 30154 tablet_replica.cc:330] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: stopping tablet replica
I20240501 19:57:39.966069 30154 raft_consensus.cc:2229] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:39.966269 30154 raft_consensus.cc:2258] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:39.967057 30154 ts_tablet_manager.cc:1875] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:39.968917 30154 ts_tablet_manager.cc:1888] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20240501 19:57:39.968989 30154 log.cc:1196] T 5fe56fe5bfe2443c97aa252754a24aa6 P 77ff839c08d0479993ba43e53980437b: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root/wals/5fe56fe5bfe2443c97aa252754a24aa6
I20240501 19:57:39.969476 29196 catalog_manager.cc:4830] TS 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839): tablet 5fe56fe5bfe2443c97aa252754a24aa6 (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:39.999933 30070 raft_consensus.cc:1050] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b: attempting to promote NON_VOTER ba2031c7573846baa1c0190d340bb485 to VOTER
I20240501 19:57:40.000363 30070 consensus_queue.cc:237] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:40.001477 29387 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:40.001569 29309 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:40.001829 29783 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485 [term 1 LEARNER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:57:40.001866 30137 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:40.002014 30136 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:40.002542 30137 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:40.004002 30156 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:40.004282 29387 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:40.004647 29783 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:40.005980 29309 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:40.006800 29207 catalog_manager.cc:5484] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb reported cstate change: config changed from index 28 to 29, ba2031c7573846baa1c0190d340bb485 (127.25.83.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:40.009534 29458 consensus_queue.cc:237] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:40.010643 29310 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:40.010716 29783 raft_consensus.cc:1261] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20240501 19:57:40.011022 30137 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:40.011154 30070 consensus_queue.cc:1035] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20240501 19:57:40.012478 30136 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:40.013278 29783 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:40.013639 29196 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0d7f921a55e437eaaddf7d276c8de0c with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:40.014082 29310 raft_consensus.cc:2932] T c0d7f921a55e437eaaddf7d276c8de0c P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } }
I20240501 19:57:40.014573 29205 catalog_manager.cc:5484] T c0d7f921a55e437eaaddf7d276c8de0c P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index 29 to 30, VOTER 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) evicted. New cstate: current_term: 1 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:40.016556 29376 tablet_service.cc:1514] Processing DeleteTablet for tablet c0d7f921a55e437eaaddf7d276c8de0c with delete_type TABLET_DATA_TOMBSTONED (TS 26f1ddf6a31645c0a82e697acc9253fb not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:40664
I20240501 19:57:40.016736 30146 tablet_replica.cc:330] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: stopping tablet replica
I20240501 19:57:40.016902 30146 raft_consensus.cc:2229] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:40.017074 30146 raft_consensus.cc:2258] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:40.017639 30146 ts_tablet_manager.cc:1875] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:40.019512 30146 ts_tablet_manager.cc:1888] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20240501 19:57:40.019570 30146 log.cc:1196] T c0d7f921a55e437eaaddf7d276c8de0c P 26f1ddf6a31645c0a82e697acc9253fb: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-1-root/wals/c0d7f921a55e437eaaddf7d276c8de0c
I20240501 19:57:40.019973 29194 catalog_manager.cc:4830] TS 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945): tablet c0d7f921a55e437eaaddf7d276c8de0c (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:40.066991 30087 raft_consensus.cc:979] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb: : Instructing follower 77ff839c08d0479993ba43e53980437b to start an election
I20240501 19:57:40.067129 30087 raft_consensus.cc:1067] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Signalling peer 77ff839c08d0479993ba43e53980437b to start an election
I20240501 19:57:40.067576 29458 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee"
dest_uuid: "77ff839c08d0479993ba43e53980437b"
from {username='slave'} at 127.0.0.1:42030
I20240501 19:57:40.067709 29458 raft_consensus.cc:482] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:40.067777 29458 raft_consensus.cc:3037] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:40.068746 29458 raft_consensus.cc:504] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:40.069154 29458 leader_election.cc:288] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 2 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171)
I20240501 19:57:40.069799 29387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:40.069921 29387 raft_consensus.cc:3032] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:40.069969 29387 raft_consensus.cc:729] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:40.070140 29387 consensus_queue.cc:260] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:40.070317 29387 raft_consensus.cc:3037] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:40.071307 29387 raft_consensus.cc:2454] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 2.
I20240501 19:57:40.071575 29427 leader_election.cc:302] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1ddf6a31645c0a82e697acc9253fb, 77ff839c08d0479993ba43e53980437b; no voters:
I20240501 19:57:40.071763 30136 raft_consensus.cc:2781] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:40.071867 30136 raft_consensus.cc:686] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 2 LEADER]: Becoming Leader. State: Replica: 77ff839c08d0479993ba43e53980437b, State: Running, Role: LEADER
I20240501 19:57:40.071990 30136 consensus_queue.cc:237] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:40.073285 29707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee" candidate_uuid: "77ff839c08d0479993ba43e53980437b" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "b83077a363e241c8b97248c2617f08dc"
I20240501 19:57:40.073413 29707 raft_consensus.cc:3037] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:40.073866 29202 catalog_manager.cc:5484] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b reported cstate change: term changed from 1 to 2, leader changed from 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) to 77ff839c08d0479993ba43e53980437b (127.25.83.3). New cstate: current_term: 2 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:40.074324 29707 raft_consensus.cc:2454] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77ff839c08d0479993ba43e53980437b in term 2.
I20240501 19:57:40.131822 29387 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 19:57:40.132189 30142 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:40.134439 29858 raft_consensus.cc:1261] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P cbbe61a596804ca093383d53b3a4b2ab [term 3 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20240501 19:57:40.136142 30140 consensus_queue.cc:1035] T 46f49bfbb5624aa89fcdfb1e8eaecc87 P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:40.192883 29309 raft_consensus.cc:1261] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 19:57:40.193269 30087 consensus_queue.cc:1035] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:40.195822 29458 raft_consensus.cc:1261] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20240501 19:57:40.196162 30071 consensus_queue.cc:1035] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:57:40.394690 30080 raft_consensus.cc:1050] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774: attempting to promote NON_VOTER b83077a363e241c8b97248c2617f08dc to VOTER
I20240501 19:57:40.395144 30080 consensus_queue.cc:237] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:40.396449 29783 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240501 19:57:40.396530 29707 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc [term 1 LEARNER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240501 19:57:40.396714 29458 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20240501 19:57:40.396828 30142 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:40.397008 30132 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:40.397295 30140 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:40.399282 30142 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:40.399646 29783 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:40.399734 29707 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:40.401345 29458 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:40.402297 29210 catalog_manager.cc:5484] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 reported cstate change: config changed from index 31 to 32, b83077a363e241c8b97248c2617f08dc (127.25.83.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:40.405524 29309 consensus_queue.cc:237] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } }
I20240501 19:57:40.406713 29707 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240501 19:57:40.406787 29783 raft_consensus.cc:1261] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Refusing update from remote peer 649b8f2f20324033bc021e229cff5774: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 1 index: 33. (index mismatch)
I20240501 19:57:40.407097 30132 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:40.407239 30080 consensus_queue.cc:1035] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:40.408676 30080 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:40.409189 29707 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P b83077a363e241c8b97248c2617f08dc [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:40.409327 29783 raft_consensus.cc:2932] T e0518736faac4d54bdb78737484c860a P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Committing config change with OpId 1.33: config changed from index 32 to 33, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } }
I20240501 19:57:40.410502 29195 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0518736faac4d54bdb78737484c860a with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:57:40.411207 29205 catalog_manager.cc:5484] T e0518736faac4d54bdb78737484c860a P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index 32 to 33, VOTER 77ff839c08d0479993ba43e53980437b (127.25.83.3) evicted. New cstate: current_term: 1 leader_uuid: "649b8f2f20324033bc021e229cff5774" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:40.413358 29454 tablet_service.cc:1514] Processing DeleteTablet for tablet e0518736faac4d54bdb78737484c860a with delete_type TABLET_DATA_TOMBSTONED (TS 77ff839c08d0479993ba43e53980437b not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:42002
I20240501 19:57:40.413560 30154 tablet_replica.cc:330] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: stopping tablet replica
I20240501 19:57:40.413692 30154 raft_consensus.cc:2229] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:57:40.413880 30154 raft_consensus.cc:2258] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:57:40.414484 30154 ts_tablet_manager.cc:1875] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:57:40.416077 30154 ts_tablet_manager.cc:1888] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.32
I20240501 19:57:40.416138 30154 log.cc:1196] T e0518736faac4d54bdb78737484c860a P 77ff839c08d0479993ba43e53980437b: Deleting WAL directory at /tmp/dist-test-taskmTnAKC/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1714593357002142-25932-0/minicluster-data/ts-2-root/wals/e0518736faac4d54bdb78737484c860a
I20240501 19:57:40.416554 29196 catalog_manager.cc:4830] TS 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839): tablet e0518736faac4d54bdb78737484c860a (table test-workload [id=9750aeea18d542b4b06c0492a5d30987]) successfully deleted
I20240501 19:57:40.421140 29387 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Refusing update from remote peer b83077a363e241c8b97248c2617f08dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 19:57:40.421429 30115 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:40.423476 29458 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Refusing update from remote peer b83077a363e241c8b97248c2617f08dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20240501 19:57:40.423960 30115 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:40.557588 29387 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 19:57:40.558058 30136 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:40.560580 29707 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20240501 19:57:40.560932 30137 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20240501 19:57:41.418272 29783 consensus_queue.cc:237] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
I20240501 19:57:41.419742 29309 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer ba2031c7573846baa1c0190d340bb485: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 19:57:41.419742 29458 raft_consensus.cc:1261] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer ba2031c7573846baa1c0190d340bb485: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 19:57:41.420152 30189 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:41.420329 30109 consensus_queue.cc:1035] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:41.421842 30189 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:41.422281 29458 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:41.423295 29309 raft_consensus.cc:2932] T 9967547056d848d1b37aff3f6b4abe86 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:41.424767 29202 catalog_manager.cc:5484] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 reported cstate change: config changed from index 31 to 33, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New cstate: current_term: 2 leader_uuid: "ba2031c7573846baa1c0190d340bb485" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:57:41.425575 29749 consensus_peers.cc:485] T 9967547056d848d1b37aff3f6b4abe86 P ba2031c7573846baa1c0190d340bb485 -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9967547056d848d1b37aff3f6b4abe86. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:41.429217 29858 consensus_queue.cc:237] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:41.430682 29387 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Refusing update from remote peer cbbe61a596804ca093383d53b3a4b2ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 19:57:41.430972 29309 raft_consensus.cc:1261] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer cbbe61a596804ca093383d53b3a4b2ab: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 19:57:41.431149 30120 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:41.431345 30200 consensus_queue.cc:1035] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:41.432976 30120 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:41.433444 29309 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:41.435343 29205 catalog_manager.cc:5484] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab reported cstate change: config changed from index 31 to 33, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New cstate: current_term: 2 leader_uuid: "cbbe61a596804ca093383d53b3a4b2ab" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:57:41.435765 29822 consensus_peers.cc:485] T 2803395236b144059af4f49d49e8c7e3 P cbbe61a596804ca093383d53b3a4b2ab -> Peer b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171): Couldn't send request to peer b83077a363e241c8b97248c2617f08dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2803395236b144059af4f49d49e8c7e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:41.436134 29387 raft_consensus.cc:2932] T 2803395236b144059af4f49d49e8c7e3 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 31 to 33, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: false } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:41.439858 29707 consensus_queue.cc:237] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:41.441500 29387 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Refusing update from remote peer b83077a363e241c8b97248c2617f08dc: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 19:57:41.441843 30208 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:41.441900 29458 raft_consensus.cc:1261] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Refusing update from remote peer b83077a363e241c8b97248c2617f08dc: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20240501 19:57:41.442238 30207 consensus_queue.cc:1035] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20240501 19:57:41.445420 29671 consensus_peers.cc:485] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc -> Peer ba2031c7573846baa1c0190d340bb485 (127.25.83.5:41139): Couldn't send request to peer ba2031c7573846baa1c0190d340bb485. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b886671d00534706ad4fc5f6023c1dd0. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:41.445613 30207 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:41.445911 29458 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 77ff839c08d0479993ba43e53980437b [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:41.446516 29387 raft_consensus.cc:2932] T b886671d00534706ad4fc5f6023c1dd0 P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:41.448554 29207 catalog_manager.cc:5484] T b886671d00534706ad4fc5f6023c1dd0 P b83077a363e241c8b97248c2617f08dc reported cstate change: config changed from index 30 to 32, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New cstate: current_term: 3 leader_uuid: "b83077a363e241c8b97248c2617f08dc" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:41.449196 29387 consensus_queue.cc:237] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } }
I20240501 19:57:41.450649 29309 raft_consensus.cc:1261] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 19:57:41.451326 30180 consensus_queue.cc:1035] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
W20240501 19:57:41.452960 29353 consensus_peers.cc:485] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb -> Peer b83077a363e241c8b97248c2617f08dc (127.25.83.4:43171): Couldn't send request to peer b83077a363e241c8b97248c2617f08dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cd1acf0afe14cc2a388fcf79ddae20d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:41.453568 29458 raft_consensus.cc:1261] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20240501 19:57:41.454535 30180 consensus_queue.cc:1035] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20240501 19:57:41.455058 30180 raft_consensus.cc:2932] T 8cd1acf0afe14cc2a388fcf79ddae20d P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:41.455518 29309 raft_consensus.cc:2932] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:41.458693 29202 catalog_manager.cc:5484] T 8cd1acf0afe14cc2a388fcf79ddae20d P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index -1 to 29, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:41.458956 29464 consensus_queue.cc:237] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } }
W20240501 19:57:41.460247 29430 consensus_peers.cc:485] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b -> Peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237): Couldn't send request to peer cbbe61a596804ca093383d53b3a4b2ab. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffed8312f8034d9fa6aa423ced8117ee. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:41.460291 29707 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:41.460340 29387 raft_consensus.cc:1261] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Refusing update from remote peer 77ff839c08d0479993ba43e53980437b: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20240501 19:57:41.460520 29464 raft_consensus.cc:2932] T 8cd1acf0afe14cc2a388fcf79ddae20d P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER b83077a363e241c8b97248c2617f08dc (127.25.83.4) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: NON_VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: true } } }
I20240501 19:57:41.460619 30137 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:41.460791 30182 consensus_queue.cc:1035] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20240501 19:57:41.462935 30137 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:41.464578 29707 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:41.464722 29387 raft_consensus.cc:2932] T ffed8312f8034d9fa6aa423ced8117ee P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } } }
I20240501 19:57:41.465181 29205 catalog_manager.cc:5484] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b reported cstate change: config changed from index 30 to 32, NON_VOTER cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6) added. New cstate: current_term: 2 leader_uuid: "77ff839c08d0479993ba43e53980437b" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbbe61a596804ca093383d53b3a4b2ab" member_type: NON_VOTER last_known_addr { host: "127.25.83.6" port: 40237 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:41.466359 29387 consensus_queue.cc:237] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:41.468715 29707 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 19:57:41.468786 29309 raft_consensus.cc:1261] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Refusing update from remote peer 26f1ddf6a31645c0a82e697acc9253fb: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20240501 19:57:41.469204 30215 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:41.469416 30184 consensus_queue.cc:1035] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20240501 19:57:41.471040 30215 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
W20240501 19:57:41.472165 29353 consensus_peers.cc:485] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb -> Peer ba2031c7573846baa1c0190d340bb485 (127.25.83.5:41139): Couldn't send request to peer ba2031c7573846baa1c0190d340bb485. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5fe56fe5bfe2443c97aa252754a24aa6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:57:41.472074 29309 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:41.472579 29707 raft_consensus.cc:2932] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:41.474020 29202 catalog_manager.cc:5484] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 reported cstate change: config changed from index 32 to 33, NON_VOTER ba2031c7573846baa1c0190d340bb485 (127.25.83.5) added. New cstate: current_term: 2 leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } } }
I20240501 19:57:41.589787 29264 auto_leader_rebalancer.cc:131] leader rebalance for table test-workload
I20240501 19:57:41.591805 29309 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39fb3d9c368d4c0d95bf935f60f863e8"
dest_uuid: "649b8f2f20324033bc021e229cff5774"
mode: GRACEFUL
new_leader_uuid: "ba2031c7573846baa1c0190d340bb485"
from {username='slave'} at 127.0.0.1:40616
I20240501 19:57:41.591956 29309 raft_consensus.cc:595] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Received request to transfer leadership to TS ba2031c7573846baa1c0190d340bb485
I20240501 19:57:41.592700 29388 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6"
dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
mode: GRACEFUL
new_leader_uuid: "b83077a363e241c8b97248c2617f08dc"
from {username='slave'} at 127.0.0.1:40732
I20240501 19:57:41.592818 29388 raft_consensus.cc:595] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Received request to transfer leadership to TS b83077a363e241c8b97248c2617f08dc
I20240501 19:57:41.593508 29458 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffed8312f8034d9fa6aa423ced8117ee"
dest_uuid: "77ff839c08d0479993ba43e53980437b"
mode: GRACEFUL
new_leader_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
from {username='slave'} at 127.0.0.1:42050
I20240501 19:57:41.593606 29458 raft_consensus.cc:595] T ffed8312f8034d9fa6aa423ced8117ee P 77ff839c08d0479993ba43e53980437b [term 2 LEADER]: Received request to transfer leadership to TS 26f1ddf6a31645c0a82e697acc9253fb
I20240501 19:57:41.593920 29264 auto_leader_rebalancer.cc:379] table: test-workload, leader rebalance finish, leader transfer count: 3
I20240501 19:57:41.594038 29264 auto_leader_rebalancer.cc:437] All tables' leader rebalancing finished this round
I20240501 19:57:41.749182 30132 raft_consensus.cc:979] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774: : Instructing follower ba2031c7573846baa1c0190d340bb485 to start an election
I20240501 19:57:41.749696 30132 raft_consensus.cc:1067] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Signalling peer ba2031c7573846baa1c0190d340bb485 to start an election
I20240501 19:57:41.751067 29782 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39fb3d9c368d4c0d95bf935f60f863e8"
dest_uuid: "ba2031c7573846baa1c0190d340bb485"
from {username='slave'} at 127.0.0.1:49990
I20240501 19:57:41.751220 29782 raft_consensus.cc:482] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:41.751292 29782 raft_consensus.cc:3037] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:41.753119 29782 raft_consensus.cc:504] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:41.753708 29782 leader_election.cc:288] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [CANDIDATE]: Term 2 election: Requested vote from peers 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115), 77ff839c08d0479993ba43e53980437b (127.25.83.3:44839)
I20240501 19:57:41.754184 29458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fb3d9c368d4c0d95bf935f60f863e8" candidate_uuid: "ba2031c7573846baa1c0190d340bb485" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "77ff839c08d0479993ba43e53980437b"
I20240501 19:57:41.754335 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39fb3d9c368d4c0d95bf935f60f863e8" candidate_uuid: "ba2031c7573846baa1c0190d340bb485" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:41.754411 29458 raft_consensus.cc:3037] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:41.754417 29309 raft_consensus.cc:3032] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:57:41.755115 29309 raft_consensus.cc:729] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 649b8f2f20324033bc021e229cff5774, State: Running, Role: LEADER
I20240501 19:57:41.755321 29309 consensus_queue.cc:260] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:41.755481 29309 raft_consensus.cc:3037] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:57:41.756027 29458 raft_consensus.cc:2454] T 39fb3d9c368d4c0d95bf935f60f863e8 P 77ff839c08d0479993ba43e53980437b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba2031c7573846baa1c0190d340bb485 in term 2.
I20240501 19:57:41.756366 29748 leader_election.cc:302] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77ff839c08d0479993ba43e53980437b, ba2031c7573846baa1c0190d340bb485; no voters:
I20240501 19:57:41.756531 29309 raft_consensus.cc:2454] T 39fb3d9c368d4c0d95bf935f60f863e8 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba2031c7573846baa1c0190d340bb485 in term 2.
I20240501 19:57:41.756590 30188 raft_consensus.cc:2781] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:57:41.756668 30188 raft_consensus.cc:686] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [term 2 LEADER]: Becoming Leader. State: Replica: ba2031c7573846baa1c0190d340bb485, State: Running, Role: LEADER
I20240501 19:57:41.756801 30188 consensus_queue.cc:237] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 31, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } }
I20240501 19:57:41.758225 29205 catalog_manager.cc:5484] T 39fb3d9c368d4c0d95bf935f60f863e8 P ba2031c7573846baa1c0190d340bb485 reported cstate change: term changed from 1 to 2, leader changed from 649b8f2f20324033bc021e229cff5774 (127.25.83.1) to ba2031c7573846baa1c0190d340bb485 (127.25.83.5). New cstate: current_term: 2 leader_uuid: "ba2031c7573846baa1c0190d340bb485" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77ff839c08d0479993ba43e53980437b" member_type: VOTER last_known_addr { host: "127.25.83.3" port: 44839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:57:41.832621 30224 ts_tablet_manager.cc:902] T 8cd1acf0afe14cc2a388fcf79ddae20d P b83077a363e241c8b97248c2617f08dc: Initiating tablet copy from peer 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
F20240501 19:57:41.832865 30224 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (init.queue_time_us, init.queue_time_us) (init.run_cpu_time_us, init.run_cpu_time_us) (init.run_wall_time_us, init.run_wall_time_us) (leader-initialization.queue_time_us, leader-initialization.queue_time_us) (leader-initialization.run_cpu_time_us, leader-initialization.run_cpu_time_us) (leader-initialization.run_wall_time_us, leader-initialization.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-253365737b4c40f8a7684abb8b807fae.queue_time_us, tablet-download-pool-253365737b4c40f8a7684abb8b807fae.queue_time_us) (tablet-download-pool-253365737b4c40f8a7684abb8b807fae.run_cpu_time_us, tablet-download-pool-253365737b4c40f8a7684abb8b807fae.run_cpu_time_us) (tablet-download-pool-253365737b4c40f8a7684abb8b807fae.run_wall_time_us, tablet-download-pool-253365737b4c40f8a7684abb8b807fae.run_wall_time_us) (tablet-download-pool-2803395236b144059af4f49d49e8c7e3.queue_time_us, tablet-download-pool-2803395236b144059af4f49d49e8c7e3.queue_time_us) (tablet-download-pool-2803395236b144059af4f49d49e8c7e3.run_cpu_time_us, tablet-download-pool-2803395236b144059af4f49d49e8c7e3.run_cpu_time_us) (tablet-download-pool-2803395236b144059af4f49d49e8c7e3.run_wall_time_us, tablet-download-pool-2803395236b144059af4f49d49e8c7e3.run_wall_time_us) (tablet-download-pool-39fb3d9c368d4c0d95bf935f60f863e8.queue_time_us, tablet-download-pool-39fb3d9c368d4c0d95bf935f60f863e8.queue_time_us) (tablet-download-pool-39fb3d9c368d4c0d95bf935f60f863e8.run_cpu_time_us, tablet-download-pool-39fb3d9c368d4c0d95bf935f60f863e8.run_cpu_time_us) (tablet-download-pool-39fb3d9c368d4c0d95bf935f60f863e8.run_wall_time_us, tablet-download-pool-39fb3d9c368d4c0d95bf935f60f863e8.run_wall_time_us) (tablet-download-pool-3e50e51f9e3c4d3d83ae49f7a326551a.queue_time_us, tablet-download-pool-3e50e51f9e3c4d3d83ae49f7a326551a.queue_time_us) (tablet-download-pool-3e50e51f9e3c4d3d83ae49f7a326551a.run_cpu_time_us, tablet-download-pool-3e50e51f9e3c4d3d83ae49f7a326551a.run_cpu_time_us) (tablet-download-pool-3e50e51f9e3c4d3d83ae49f7a326551a.run_wall_time_us, tablet-download-pool-3e50e51f9e3c4d3d83ae49f7a326551a.run_wall_time_us) (tablet-download-pool-46f49bfbb5624aa89fcdfb1e8eaecc87.queue_time_us, tablet-download-pool-46f49bfbb5624aa89fcdfb1e8eaecc87.queue_time_us) (tablet-download-pool-46f49bfbb5624aa89fcdfb1e8eaecc87.run_cpu_time_us, tablet-download-pool-46f49bfbb5624aa89fcdfb1e8eaecc87.run_cpu_time_us) (tablet-download-pool-46f49bfbb5624aa89fcdfb1e8eaecc87.run_wall_time_us, tablet-download-pool-46f49bfbb5624aa89fcdfb1e8eaecc87.run_wall_time_us) (tablet-download-pool-5fe56fe5bfe2443c97aa252754a24aa6.queue_time_us, tablet-download-pool-5fe56fe5bfe2443c97aa252754a24aa6.queue_time_us) (tablet-download-pool-5fe56fe5bfe2443c97aa252754a24aa6.run_cpu_time_us, tablet-download-pool-5fe56fe5bfe2443c97aa252754a24aa6.run_cpu_time_us) (tablet-download-pool-5fe56fe5bfe2443c97aa252754a24aa6.run_wall_time_us, tablet-download-pool-5fe56fe5bfe2443c97aa252754a24aa6.run_wall_time_us) (tablet-download-pool-8cd1acf0afe14cc2a388fcf79ddae20d.queue_time_us, tablet-download-pool-8cd1acf0afe14cc2a388fcf79ddae20d.queue_time_us) (tablet-download-pool-9967547056d848d1b37aff3f6b4abe86.queue_time_us, tablet-download-pool-9967547056d848d1b37aff3f6b4abe86.queue_time_us) (tablet-download-pool-9967547056d848d1b37aff3f6b4abe86.run_cpu_time_us, tablet-download-pool-9967547056d848d1b37aff3f6b4abe86.run_cpu_time_us) (tablet-download-pool-9967547056d848d1b37aff3f6b4abe86.run_wall_time_us, tablet-download-pool-9967547056d848d1b37aff3f6b4abe86.run_wall_time_us) (tablet-download-pool-a2f89b5db8c34edaa8751a263dc818cd.queue_time_us, tablet-download-pool-a2f89b5db8c34edaa8751a263dc818cd.queue_time_us) (tablet-download-pool-a2f89b5db8c34edaa8751a263dc818cd.run_cpu_time_us, tablet-download-pool-a2f89b5db8c34edaa8751a263dc818cd.run_cpu_time_us) (tablet-download-pool-a2f89b5db8c34edaa8751a263dc818cd.run_wall_time_us, tablet-download-pool-a2f89b5db8c34edaa8751a263dc818cd.run_wall_time_us) (tablet-download-pool-b886671d00534706ad4fc5f6023c1dd0.queue_time_us, tablet-download-pool-b886671d00534706ad4fc5f6023c1dd0.queue_time_us) (tablet-download-pool-b886671d00534706ad4fc5f6023c1dd0.run_cpu_time_us, tablet-download-pool-b886671d00534706ad4fc5f6023c1dd0.run_cpu_time_us) (tablet-download-pool-b886671d00534706ad4fc5f6023c1dd0.run_wall_time_us, tablet-download-pool-b886671d00534706ad4fc5f6023c1dd0.run_wall_time_us) (tablet-download-pool-c0d7f921a55e437eaaddf7d276c8de0c.queue_time_us, tablet-download-pool-c0d7f921a55e437eaaddf7d276c8de0c.queue_time_us) (tablet-download-pool-c0d7f921a55e437eaaddf7d276c8de0c.run_cpu_time_us, tablet-download-pool-c0d7f921a55e437eaaddf7d276c8de0c.run_cpu_time_us) (tablet-download-pool-c0d7f921a55e437eaaddf7d276c8de0c.run_wall_time_us, tablet-download-pool-c0d7f921a55e437eaaddf7d276c8de0c.run_wall_time_us) (tablet-download-pool-e0518736faac4d54bdb78737484c860a.queue_time_us, tablet-download-pool-e0518736faac4d54bdb78737484c860a.queue_time_us) (tablet-download-pool-e0518736faac4d54bdb78737484c860a.run_cpu_time_us, tablet-download-pool-e0518736faac4d54bdb78737484c860a.run_cpu_time_us) (tablet-download-pool-e0518736faac4d54bdb78737484c860a.run_wall_time_us, tablet-download-pool-e0518736faac4d54bdb78737484c860a.run_wall_time_us) (tablet-download-pool-ffed8312f8034d9fa6aa423ced8117ee.queue_time_us, tablet-download-pool-ffed8312f8034d9fa6aa423ced8117ee.queue_time_us) (tablet-download-pool-ffed8312f8034d9fa6aa423ced8117ee.run_cpu_time_us, tablet-download-pool-ffed8312f8034d9fa6aa423ced8117ee.run_cpu_time_us) (tablet-download-pool-ffed8312f8034d9fa6aa423ced8117ee.run_wall_time_us, tablet-download-pool-ffed8312f8034d9fa6aa423ced8117ee.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
*** Aborted at 1714593461 (unix time) try "date -d @1714593461" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e80000654c) received by PID 25932 (TID 0x7fb57fee8700) from PID 25932; stack trace: ***
@ 0x7fb6449b7040 (unknown) at ??:0
I20240501 19:57:41.844650 30226 ts_tablet_manager.cc:902] T 5fe56fe5bfe2443c97aa252754a24aa6 P ba2031c7573846baa1c0190d340bb485: Initiating tablet copy from peer 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945)
@ 0x7fb6449b6fb7 gsignal at ??:0
@ 0x7fb6449b8921 abort at ??:0
@ 0x7fb6456bbdcd google::LogMessage::Fail() at ??:0
@ 0x7fb6456bfb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb6456bb7cc google::LogMessage::Flush() at ??:0
@ 0x7fb6456bcf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb646213044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fb6461f7f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fb6461f5efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fb64717277a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fb64717ba45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fb6471fc76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fb6472104f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fb6471fb226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fb64720ac0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb6477653cc std::function<>::operator()() at ??:0
@ 0x7fb6461fae06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb6461fb6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 19:57:41.925614 30184 raft_consensus.cc:979] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb: : Instructing follower b83077a363e241c8b97248c2617f08dc to start an election
I20240501 19:57:41.925711 30184 raft_consensus.cc:1067] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Signalling peer b83077a363e241c8b97248c2617f08dc to start an election
I20240501 19:57:41.926095 29707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6"
dest_uuid: "b83077a363e241c8b97248c2617f08dc"
from {username='slave'} at 127.0.0.1:48612
I20240501 19:57:41.926221 29707 raft_consensus.cc:482] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:57:41.926275 29707 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:41.927407 29707 raft_consensus.cc:504] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:41.927811 29707 leader_election.cc:288] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [CANDIDATE]: Term 3 election: Requested vote from peers 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2:32945), 649b8f2f20324033bc021e229cff5774 (127.25.83.1:46115)
I20240501 19:57:41.928242 29309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6" candidate_uuid: "b83077a363e241c8b97248c2617f08dc" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: true dest_uuid: "649b8f2f20324033bc021e229cff5774"
I20240501 19:57:41.928535 29309 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:41.928748 29388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fe56fe5bfe2443c97aa252754a24aa6" candidate_uuid: "b83077a363e241c8b97248c2617f08dc" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: true dest_uuid: "26f1ddf6a31645c0a82e697acc9253fb"
I20240501 19:57:41.928879 29388 raft_consensus.cc:3032] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:57:41.928938 29388 raft_consensus.cc:729] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 26f1ddf6a31645c0a82e697acc9253fb, State: Running, Role: LEADER
I20240501 19:57:41.929114 29388 consensus_queue.cc:260] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
@ 0x7fb6461fd03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 19:57:41.929344 29388 raft_consensus.cc:3037] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:57:41.929430 29309 raft_consensus.cc:2454] T 5fe56fe5bfe2443c97aa252754a24aa6 P 649b8f2f20324033bc021e229cff5774 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83077a363e241c8b97248c2617f08dc in term 3.
I20240501 19:57:41.930446 29388 raft_consensus.cc:2454] T 5fe56fe5bfe2443c97aa252754a24aa6 P 26f1ddf6a31645c0a82e697acc9253fb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b83077a363e241c8b97248c2617f08dc in term 3.
I20240501 19:57:41.930701 29671 leader_election.cc:302] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649b8f2f20324033bc021e229cff5774, b83077a363e241c8b97248c2617f08dc; no voters:
I20240501 19:57:41.930871 30208 raft_consensus.cc:2781] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:57:41.930963 30208 raft_consensus.cc:686] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [term 3 LEADER]: Becoming Leader. State: Replica: b83077a363e241c8b97248c2617f08dc, State: Running, Role: LEADER
I20240501 19:57:41.931105 30208 consensus_queue.cc:237] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } }
I20240501 19:57:41.933507 29205 catalog_manager.cc:5484] T 5fe56fe5bfe2443c97aa252754a24aa6 P b83077a363e241c8b97248c2617f08dc reported cstate change: term changed from 2 to 3, leader changed from 26f1ddf6a31645c0a82e697acc9253fb (127.25.83.2) to b83077a363e241c8b97248c2617f08dc (127.25.83.4). New cstate: current_term: 3 leader_uuid: "b83077a363e241c8b97248c2617f08dc" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "26f1ddf6a31645c0a82e697acc9253fb" member_type: VOTER last_known_addr { host: "127.25.83.2" port: 32945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "649b8f2f20324033bc021e229cff5774" member_type: VOTER last_known_addr { host: "127.25.83.1" port: 46115 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b83077a363e241c8b97248c2617f08dc" member_type: VOTER last_known_addr { host: "127.25.83.4" port: 43171 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba2031c7573846baa1c0190d340bb485" member_type: NON_VOTER last_known_addr { host: "127.25.83.5" port: 41139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:57:41.934633 30246 ts_tablet_manager.cc:902] T 2803395236b144059af4f49d49e8c7e3 P b83077a363e241c8b97248c2617f08dc: Initiating tablet copy from peer cbbe61a596804ca093383d53b3a4b2ab (127.25.83.6:40237)
@ 0x7fb6477653cc std::function<>::operator()() at ??:0
@ 0x7fb6461ea807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb6429ec6db start_thread at ??:0
@ 0x7fb644a9971f clone at ??:0