[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250111 02:51:12.563122 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:44145,127.0.103.61:33967,127.0.103.60:35137
I20250111 02:51:12.564945 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250111 02:51:12.566118 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:12.576848 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:12.578097 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:12.579145 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:12.580021 412 server_base.cc:1034] running on GCE node
I20250111 02:51:12.580750 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:12.580886 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:12.580950 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563872580935 us; error 0 us; skew 500 ppm
I20250111 02:51:12.581246 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:12.583889 412 webserver.cc:458] Webserver started at http://127.0.103.62:36445/ using document root <none> and password file <none>
I20250111 02:51:12.584580 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:12.584666 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:12.584959 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:12.588717 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "869e46bd0ed549fe9f42900c453f4fdd"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:12.594168 412 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20250111 02:51:12.597186 425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:12.598580 412 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250111 02:51:12.598738 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "869e46bd0ed549fe9f42900c453f4fdd"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:12.598901 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:12.676538 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:12.677397 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250111 02:51:12.677726 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:12.697027 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:44145
I20250111 02:51:12.697499 476 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:44145 every 8 connection(s)
I20250111 02:51:12.699133 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:51:12.702571 477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250111 02:51:12.702541 479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:12.703665 480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:12.704641 482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:12.705518 412 server_base.cc:1034] running on GCE node
I20250111 02:51:12.705804 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:12.705904 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:12.705984 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563872705945 us; error 0 us; skew 500 ppm
I20250111 02:51:12.706158 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:12.708070 412 webserver.cc:458] Webserver started at http://127.0.103.61:43817/ using document root <none> and password file <none>
I20250111 02:51:12.708313 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:12.708415 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:12.708562 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:12.709281 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root/instance:
uuid: "9081218c6e0d483bbbec242612898c58"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:12.711920 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250111 02:51:12.713537 487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:12.713937 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:12.714037 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root
uuid: "9081218c6e0d483bbbec242612898c58"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:12.714190 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:12.713755 477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:12.729660 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:12.731038 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:12.732380 477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:51:12.733554 429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.61:33967: connect: Connection refused (error 111)
W20250111 02:51:12.735569 477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.61:33967: Network error: Client connection negotiation failed: client connection to 127.0.103.61:33967: connect: Connection refused (error 111)
I20250111 02:51:12.748426 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.61:33967
I20250111 02:51:12.748459 541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.61:33967 every 8 connection(s)
I20250111 02:51:12.750310 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:51:12.750397 542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:12.753376 542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:51:12.753891 544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:12.754640 545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:12.756537 547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:12.757313 412 server_base.cc:1034] running on GCE node
I20250111 02:51:12.757601 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:12.757668 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:12.757768 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563872757727 us; error 0 us; skew 500 ppm
I20250111 02:51:12.757963 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:12.759879 412 webserver.cc:458] Webserver started at http://127.0.103.60:41463/ using document root <none> and password file <none>
I20250111 02:51:12.760159 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:12.760291 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:12.760438 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:12.761142 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root/instance:
uuid: "3bd6c433dc994753a1c43b0f71edc787"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:12.762121 542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:12.764295 412 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250111 02:51:12.766139 554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:12.766582 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250111 02:51:12.766680 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root
uuid: "3bd6c433dc994753a1c43b0f71edc787"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:12.766842 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:12.769443 542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:51:12.771364 542 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.60:35137: Network error: Client connection negotiation failed: client connection to 127.0.103.60:35137: connect: Connection refused (error 111)
I20250111 02:51:12.782609 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:12.783115 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:12.784857 477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } attempt: 1
I20250111 02:51:12.792629 477 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:51:12.795125 477 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.60:35137: Network error: Client connection negotiation failed: client connection to 127.0.103.60:35137: connect: Connection refused (error 111)
I20250111 02:51:12.801193 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.60:35137
I20250111 02:51:12.801271 606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.60:35137 every 8 connection(s)
I20250111 02:51:12.802165 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:51:12.803056 607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:12.805303 607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:12.811478 607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:12.816068 607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:12.823580 542 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } attempt: 1
I20250111 02:51:12.825583 607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: Bootstrap starting.
I20250111 02:51:12.829620 607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:12.831195 607 log.cc:826] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: Log is configured to *not* fsync() on all Append() calls
I20250111 02:51:12.831982 542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58: Bootstrap starting.
I20250111 02:51:12.833541 542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:12.834165 607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: No bootstrap required, opened a new log
I20250111 02:51:12.835397 542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58: No bootstrap required, opened a new log
I20250111 02:51:12.838310 477 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } attempt: 1
I20250111 02:51:12.842448 542 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
I20250111 02:51:12.842468 607 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
I20250111 02:51:12.842900 542 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:12.842985 607 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:12.843030 542 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9081218c6e0d483bbbec242612898c58, State: Initialized, Role: FOLLOWER
I20250111 02:51:12.843106 607 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd6c433dc994753a1c43b0f71edc787, State: Initialized, Role: FOLLOWER
I20250111 02:51:12.843822 607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
I20250111 02:51:12.843822 542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
I20250111 02:51:12.844841 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:12.845152 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:12.845707 614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:12.845986 614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:12.846302 542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:12.846333 607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:12.847287 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd: Bootstrap starting.
I20250111 02:51:12.849874 477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:12.855994 477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd: No bootstrap required, opened a new log
I20250111 02:51:12.857053 477 raft_consensus.cc:357] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
I20250111 02:51:12.857317 477 raft_consensus.cc:383] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:12.857421 477 raft_consensus.cc:738] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869e46bd0ed549fe9f42900c453f4fdd, State: Initialized, Role: FOLLOWER
I20250111 02:51:12.857839 477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
I20250111 02:51:12.858877 477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:12.858729 638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:12.859253 638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: This master's current role is: FOLLOWER
W20250111 02:51:12.863687 636 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:51:12.863978 636 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:51:12.863986 637 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:51:12.864127 637 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:51:12.869048 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250111 02:51:12.869179 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250111 02:51:12.870193 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:12.870620 650 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:51:12.870715 650 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:51:12.874845 651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:12.876121 652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:12.877003 654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:12.880100 412 server_base.cc:1034] running on GCE node
I20250111 02:51:12.880407 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:12.880519 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:12.880584 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563872880560 us; error 0 us; skew 500 ppm
I20250111 02:51:12.880785 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:12.882566 412 webserver.cc:458] Webserver started at http://127.0.103.1:38943/ using document root <none> and password file <none>
I20250111 02:51:12.882845 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:12.882934 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:12.883096 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:12.884339 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "c218f3b887f745e087e658e039accbef"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:12.887518 412 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250111 02:51:12.889315 659 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:12.889707 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250111 02:51:12.889806 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "c218f3b887f745e087e658e039accbef"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:12.889962 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:12.896308 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:12.896858 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:12.898041 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:12.899461 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:12.899581 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:12.899688 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:12.899770 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:12.922947 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:42057
I20250111 02:51:12.922964 721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:42057 every 8 connection(s)
W20250111 02:51:12.930321 431 tablet.cc:2367] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:51:12.938762 723 heartbeater.cc:346] Connected to a master server at 127.0.103.62:44145
I20250111 02:51:12.939116 723 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:12.940531 723 heartbeater.cc:510] Master 127.0.103.62:44145 requested a full tablet report, sending...
I20250111 02:51:12.943629 442 ts_manager.cc:194] Registered new tserver with Master: c218f3b887f745e087e658e039accbef (127.0.103.1:42057)
I20250111 02:51:12.950974 722 heartbeater.cc:346] Connected to a master server at 127.0.103.60:35137
I20250111 02:51:12.951164 722 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:12.951216 638 raft_consensus.cc:491] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:12.951526 638 raft_consensus.cc:513] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
I20250111 02:51:12.951893 722 heartbeater.cc:510] Master 127.0.103.60:35137 requested a full tablet report, sending...
I20250111 02:51:12.953090 572 ts_manager.cc:194] Registered new tserver with Master: c218f3b887f745e087e658e039accbef (127.0.103.1:42057)
I20250111 02:51:12.953483 638 leader_election.cc:290] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9081218c6e0d483bbbec242612898c58 (127.0.103.61:33967), 3bd6c433dc994753a1c43b0f71edc787 (127.0.103.60:35137)
I20250111 02:51:12.953796 724 heartbeater.cc:346] Connected to a master server at 127.0.103.61:33967
I20250111 02:51:12.953967 724 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:12.954872 517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "869e46bd0ed549fe9f42900c453f4fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081218c6e0d483bbbec242612898c58" is_pre_election: true
I20250111 02:51:12.955054 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "869e46bd0ed549fe9f42900c453f4fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd6c433dc994753a1c43b0f71edc787" is_pre_election: true
I20250111 02:51:12.955216 724 heartbeater.cc:510] Master 127.0.103.61:33967 requested a full tablet report, sending...
I20250111 02:51:12.955392 517 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869e46bd0ed549fe9f42900c453f4fdd in term 0.
I20250111 02:51:12.956133 429 leader_election.cc:304] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869e46bd0ed549fe9f42900c453f4fdd, 9081218c6e0d483bbbec242612898c58; no voters:
I20250111 02:51:12.956395 507 ts_manager.cc:194] Registered new tserver with Master: c218f3b887f745e087e658e039accbef (127.0.103.1:42057)
I20250111 02:51:12.956421 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:51:12.955396 582 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869e46bd0ed549fe9f42900c453f4fdd in term 0.
I20250111 02:51:12.957077 638 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:12.957226 638 raft_consensus.cc:491] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:12.957333 638 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:12.959985 638 raft_consensus.cc:513] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
I20250111 02:51:12.960932 517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "869e46bd0ed549fe9f42900c453f4fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9081218c6e0d483bbbec242612898c58"
I20250111 02:51:12.961043 582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "869e46bd0ed549fe9f42900c453f4fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd6c433dc994753a1c43b0f71edc787"
I20250111 02:51:12.961125 517 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:12.961179 582 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:51:12.961611 729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:12.964046 582 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869e46bd0ed549fe9f42900c453f4fdd in term 1.
I20250111 02:51:12.964385 517 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869e46bd0ed549fe9f42900c453f4fdd in term 1.
I20250111 02:51:12.964452 427 leader_election.cc:304] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd6c433dc994753a1c43b0f71edc787, 869e46bd0ed549fe9f42900c453f4fdd; no voters:
I20250111 02:51:12.965262 638 leader_election.cc:290] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [CANDIDATE]: Term 1 election: Requested vote from peers 9081218c6e0d483bbbec242612898c58 (127.0.103.61:33967), 3bd6c433dc994753a1c43b0f71edc787 (127.0.103.60:35137)
I20250111 02:51:12.965492 638 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:12.965679 638 raft_consensus.cc:695] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 1 LEADER]: Becoming Leader. State: Replica: 869e46bd0ed549fe9f42900c453f4fdd, State: Running, Role: LEADER
I20250111 02:51:12.966478 638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } }
W20250111 02:51:12.976912 730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:12.977823 731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869e46bd0ed549fe9f42900c453f4fdd. Latest consensus state: current_term: 1 leader_uuid: "869e46bd0ed549fe9f42900c453f4fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:12.978324 731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:12.982587 736 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:51:12.986479 736 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250111 02:51:12.987128 734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:12.988224 412 server_base.cc:1034] running on GCE node
I20250111 02:51:12.988621 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:12.988713 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:12.988801 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563872988760 us; error 0 us; skew 500 ppm
I20250111 02:51:12.989005 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:12.994807 517 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 1 FOLLOWER]: Refusing update from remote peer 869e46bd0ed549fe9f42900c453f4fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:51:12.994819 582 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 1 FOLLOWER]: Refusing update from remote peer 869e46bd0ed549fe9f42900c453f4fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:51:12.995726 731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:12.995915 412 webserver.cc:458] Webserver started at http://127.0.103.2:35429/ using document root <none> and password file <none>
I20250111 02:51:12.996189 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:12.996325 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:12.996488 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:12.996511 731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:12.997325 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "e0b024e7912647ca9880b8a62aaf698b"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:13.009459 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869e46bd0ed549fe9f42900c453f4fdd. Latest consensus state: current_term: 1 leader_uuid: "869e46bd0ed549fe9f42900c453f4fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:13.009730 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:13.012396 738 mvcc.cc:204] Tried to move back new op lower bound from 7112965623776604160 to 7112965623677902848. Current Snapshot: MvccSnapshot[applied={T|T < 7112965623776604160}]
I20250111 02:51:13.023519 638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869e46bd0ed549fe9f42900c453f4fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:13.023782 638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:13.025913 412 fs_manager.cc:696] Time spent creating directory manager: real 0.028s user 0.012s sys 0.017s
I20250111 02:51:13.026041 736 catalog_manager.cc:1348] Generated new cluster ID: f4028b03f07a43e496df154be537e5aa
I20250111 02:51:13.026158 736 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:51:13.031230 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869e46bd0ed549fe9f42900c453f4fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:13.031440 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:13.032275 614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869e46bd0ed549fe9f42900c453f4fdd. Latest consensus state: current_term: 1 leader_uuid: "869e46bd0ed549fe9f42900c453f4fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:13.032604 614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:13.033829 731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "869e46bd0ed549fe9f42900c453f4fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:13.034096 731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:13.036433 614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "869e46bd0ed549fe9f42900c453f4fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869e46bd0ed549fe9f42900c453f4fdd" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 44145 } } peers { permanent_uuid: "9081218c6e0d483bbbec242612898c58" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 33967 } } peers { permanent_uuid: "3bd6c433dc994753a1c43b0f71edc787" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 35137 } } }
I20250111 02:51:13.036654 614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:13.037501 747 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:13.037964 412 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.000s sys 0.010s
I20250111 02:51:13.038089 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "e0b024e7912647ca9880b8a62aaf698b"
format_stamp: "Formatted at 2025-01-11 02:51:12 on dist-test-slave-4xrj"
I20250111 02:51:13.038702 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:13.064610 736 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:51:13.067117 736 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:51:13.079653 736 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd: Generated new TSK 0
I20250111 02:51:13.080715 736 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:51:13.082968 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:13.083564 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:13.084714 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:13.085604 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:13.085690 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:13.085831 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:13.085920 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:13.106932 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:34963
I20250111 02:51:13.106952 810 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:34963 every 8 connection(s)
I20250111 02:51:13.129480 811 heartbeater.cc:346] Connected to a master server at 127.0.103.60:35137
I20250111 02:51:13.129642 811 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:13.130199 811 heartbeater.cc:510] Master 127.0.103.60:35137 requested a full tablet report, sending...
I20250111 02:51:13.131841 812 heartbeater.cc:346] Connected to a master server at 127.0.103.62:44145
I20250111 02:51:13.131973 812 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:13.132071 572 ts_manager.cc:194] Registered new tserver with Master: e0b024e7912647ca9880b8a62aaf698b (127.0.103.2:34963)
I20250111 02:51:13.132562 812 heartbeater.cc:510] Master 127.0.103.62:44145 requested a full tablet report, sending...
I20250111 02:51:13.133373 442 ts_manager.cc:194] Registered new tserver with Master: e0b024e7912647ca9880b8a62aaf698b (127.0.103.2:34963)
I20250111 02:51:13.135804 442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55492
I20250111 02:51:13.138674 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:51:13.139482 813 heartbeater.cc:346] Connected to a master server at 127.0.103.61:33967
I20250111 02:51:13.139657 813 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:13.140059 813 heartbeater.cc:510] Master 127.0.103.61:33967 requested a full tablet report, sending...
I20250111 02:51:13.141666 507 ts_manager.cc:194] Registered new tserver with Master: e0b024e7912647ca9880b8a62aaf698b (127.0.103.2:34963)
W20250111 02:51:13.144210 817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:13.144977 818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:13.147786 412 server_base.cc:1034] running on GCE node
W20250111 02:51:13.148867 820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:13.149273 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:13.149370 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:13.149435 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563873149409 us; error 0 us; skew 500 ppm
I20250111 02:51:13.149590 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:13.151121 412 webserver.cc:458] Webserver started at http://127.0.103.3:33083/ using document root <none> and password file <none>
I20250111 02:51:13.151377 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:13.151471 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:13.151638 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:13.152271 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "78973ac01961460faf125c074bf617ad"
format_stamp: "Formatted at 2025-01-11 02:51:13 on dist-test-slave-4xrj"
I20250111 02:51:13.155730 412 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250111 02:51:13.157472 825 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:13.157892 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250111 02:51:13.158015 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "78973ac01961460faf125c074bf617ad"
format_stamp: "Formatted at 2025-01-11 02:51:13 on dist-test-slave-4xrj"
I20250111 02:51:13.158143 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:13.168457 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:13.169641 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:13.170347 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:13.171479 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:13.171579 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:13.171674 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:13.171751 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:13.192306 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:43329
I20250111 02:51:13.192376 887 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:43329 every 8 connection(s)
I20250111 02:51:13.206070 888 heartbeater.cc:346] Connected to a master server at 127.0.103.60:35137
I20250111 02:51:13.206236 888 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:13.206854 888 heartbeater.cc:510] Master 127.0.103.60:35137 requested a full tablet report, sending...
I20250111 02:51:13.208163 572 ts_manager.cc:194] Registered new tserver with Master: 78973ac01961460faf125c074bf617ad (127.0.103.3:43329)
I20250111 02:51:13.214107 890 heartbeater.cc:346] Connected to a master server at 127.0.103.61:33967
I20250111 02:51:13.214272 890 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:13.214697 890 heartbeater.cc:510] Master 127.0.103.61:33967 requested a full tablet report, sending...
I20250111 02:51:13.215727 507 ts_manager.cc:194] Registered new tserver with Master: 78973ac01961460faf125c074bf617ad (127.0.103.3:43329)
I20250111 02:51:13.216202 889 heartbeater.cc:346] Connected to a master server at 127.0.103.62:44145
I20250111 02:51:13.216336 889 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:13.216782 889 heartbeater.cc:510] Master 127.0.103.62:44145 requested a full tablet report, sending...
I20250111 02:51:13.217515 442 ts_manager.cc:194] Registered new tserver with Master: 78973ac01961460faf125c074bf617ad (127.0.103.3:43329)
I20250111 02:51:13.218042 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01413525s
I20250111 02:51:13.218788 442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55498
I20250111 02:51:13.866528 637 catalog_manager.cc:1260] Loaded cluster ID: f4028b03f07a43e496df154be537e5aa
I20250111 02:51:13.866528 636 catalog_manager.cc:1260] Loaded cluster ID: f4028b03f07a43e496df154be537e5aa
I20250111 02:51:13.866704 637 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: loading cluster ID for follower catalog manager: success
I20250111 02:51:13.866770 636 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58: loading cluster ID for follower catalog manager: success
I20250111 02:51:13.869879 637 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: acquiring CA information for follower catalog manager: success
I20250111 02:51:13.869897 636 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58: acquiring CA information for follower catalog manager: success
I20250111 02:51:13.871618 637 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:51:13.871618 636 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:51:13.946995 442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55482
I20250111 02:51:14.138342 812 heartbeater.cc:502] Master 127.0.103.62:44145 was elected leader, sending a full tablet report...
I20250111 02:51:14.220227 889 heartbeater.cc:502] Master 127.0.103.62:44145 was elected leader, sending a full tablet report...
I20250111 02:51:14.869022 649 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:14.948243 723 heartbeater.cc:502] Master 127.0.103.62:44145 was elected leader, sending a full tablet report...
I20250111 02:51:15.273415 412 test_util.cc:274] Using random seed: 1223318190
I20250111 02:51:15.307292 442 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44566:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:51:15.339764 853 tablet_service.cc:1467] Processing CreateTablet for tablet db1bfd7d7e5a484b8b8aee0676d8a99b (DEFAULT_TABLE table=test-workload [id=ca1b71107dce43d7b53feda938ba82e3]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:51:15.340787 853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db1bfd7d7e5a484b8b8aee0676d8a99b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:15.343708 776 tablet_service.cc:1467] Processing CreateTablet for tablet 8e77959392c045b987a1670857b13c0f (DEFAULT_TABLE table=test-workload [id=ca1b71107dce43d7b53feda938ba82e3]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:51:15.343844 686 tablet_service.cc:1467] Processing CreateTablet for tablet 42a9570933174bd58ed53f1ab5df1144 (DEFAULT_TABLE table=test-workload [id=ca1b71107dce43d7b53feda938ba82e3]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:51:15.344269 776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e77959392c045b987a1670857b13c0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:15.344360 686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42a9570933174bd58ed53f1ab5df1144. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:15.343674 687 tablet_service.cc:1467] Processing CreateTablet for tablet 1f8fe93e7ff0468cb2dc34c3759302f8 (DEFAULT_TABLE table=test-workload [id=ca1b71107dce43d7b53feda938ba82e3]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:51:15.345963 687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f8fe93e7ff0468cb2dc34c3759302f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:15.350327 924 tablet_bootstrap.cc:492] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad: Bootstrap starting.
I20250111 02:51:15.351405 925 tablet_bootstrap.cc:492] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef: Bootstrap starting.
I20250111 02:51:15.352838 924 tablet_bootstrap.cc:654] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:15.354064 925 tablet_bootstrap.cc:654] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:15.354708 926 tablet_bootstrap.cc:492] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b: Bootstrap starting.
I20250111 02:51:15.356557 924 tablet_bootstrap.cc:492] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad: No bootstrap required, opened a new log
I20250111 02:51:15.356707 924 ts_tablet_manager.cc:1397] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20250111 02:51:15.357308 924 raft_consensus.cc:357] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78973ac01961460faf125c074bf617ad" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43329 } }
I20250111 02:51:15.357462 924 raft_consensus.cc:383] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:15.357585 924 raft_consensus.cc:738] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78973ac01961460faf125c074bf617ad, State: Initialized, Role: FOLLOWER
I20250111 02:51:15.357649 926 tablet_bootstrap.cc:654] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:15.357815 924 consensus_queue.cc:260] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78973ac01961460faf125c074bf617ad" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43329 } }
I20250111 02:51:15.358194 924 raft_consensus.cc:397] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:15.358342 924 raft_consensus.cc:491] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:15.358489 924 raft_consensus.cc:3054] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:15.359081 925 tablet_bootstrap.cc:492] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef: No bootstrap required, opened a new log
I20250111 02:51:15.359230 925 ts_tablet_manager.cc:1397] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20250111 02:51:15.360245 926 tablet_bootstrap.cc:492] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b: No bootstrap required, opened a new log
I20250111 02:51:15.360216 925 raft_consensus.cc:357] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } }
I20250111 02:51:15.360378 926 ts_tablet_manager.cc:1397] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250111 02:51:15.360409 925 raft_consensus.cc:383] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:15.360553 925 raft_consensus.cc:738] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c218f3b887f745e087e658e039accbef, State: Initialized, Role: FOLLOWER
I20250111 02:51:15.360800 925 consensus_queue.cc:260] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } }
I20250111 02:51:15.361064 925 raft_consensus.cc:397] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:15.361022 926 raft_consensus.cc:357] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b024e7912647ca9880b8a62aaf698b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34963 } }
I20250111 02:51:15.361188 925 raft_consensus.cc:491] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:15.361227 926 raft_consensus.cc:383] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:15.361299 925 raft_consensus.cc:3054] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:15.361336 926 raft_consensus.cc:738] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b024e7912647ca9880b8a62aaf698b, State: Initialized, Role: FOLLOWER
I20250111 02:51:15.361534 926 consensus_queue.cc:260] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b024e7912647ca9880b8a62aaf698b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34963 } }
I20250111 02:51:15.361711 926 raft_consensus.cc:397] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:15.361699 924 raft_consensus.cc:513] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78973ac01961460faf125c074bf617ad" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43329 } }
I20250111 02:51:15.361825 926 raft_consensus.cc:491] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:15.361956 926 raft_consensus.cc:3054] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:15.361994 924 leader_election.cc:304] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 78973ac01961460faf125c074bf617ad; no voters:
I20250111 02:51:15.364217 926 raft_consensus.cc:513] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b024e7912647ca9880b8a62aaf698b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34963 } }
I20250111 02:51:15.364436 926 leader_election.cc:304] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0b024e7912647ca9880b8a62aaf698b; no voters:
I20250111 02:51:15.364907 926 leader_election.cc:290] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:15.364831 925 raft_consensus.cc:513] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } }
I20250111 02:51:15.365151 925 leader_election.cc:304] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c218f3b887f745e087e658e039accbef; no voters:
I20250111 02:51:15.365242 931 raft_consensus.cc:2798] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:15.368837 924 leader_election.cc:290] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:15.369031 930 raft_consensus.cc:2798] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:15.371821 925 leader_election.cc:290] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:15.371989 933 raft_consensus.cc:2798] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:15.373935 924 ts_tablet_manager.cc:1428] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
I20250111 02:51:15.374606 933 raft_consensus.cc:695] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 1 LEADER]: Becoming Leader. State: Replica: c218f3b887f745e087e658e039accbef, State: Running, Role: LEADER
I20250111 02:51:15.375005 933 consensus_queue.cc:237] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } }
I20250111 02:51:15.376933 930 raft_consensus.cc:695] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 1 LEADER]: Becoming Leader. State: Replica: 78973ac01961460faf125c074bf617ad, State: Running, Role: LEADER
I20250111 02:51:15.377310 930 consensus_queue.cc:237] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78973ac01961460faf125c074bf617ad" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43329 } }
I20250111 02:51:15.381071 926 ts_tablet_manager.cc:1428] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b: Time spent starting tablet: real 0.021s user 0.007s sys 0.008s
I20250111 02:51:15.382792 931 raft_consensus.cc:695] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 1 LEADER]: Becoming Leader. State: Replica: e0b024e7912647ca9880b8a62aaf698b, State: Running, Role: LEADER
I20250111 02:51:15.383158 931 consensus_queue.cc:237] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b024e7912647ca9880b8a62aaf698b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34963 } }
I20250111 02:51:15.386305 442 catalog_manager.cc:5526] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef reported cstate change: term changed from 0 to 1, leader changed from <none> to c218f3b887f745e087e658e039accbef (127.0.103.1). New cstate: current_term: 1 leader_uuid: "c218f3b887f745e087e658e039accbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:15.391857 925 ts_tablet_manager.cc:1428] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef: Time spent starting tablet: real 0.032s user 0.006s sys 0.026s
I20250111 02:51:15.393023 925 tablet_bootstrap.cc:492] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef: Bootstrap starting.
I20250111 02:51:15.393265 441 catalog_manager.cc:5526] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b reported cstate change: term changed from 0 to 1, leader changed from <none> to e0b024e7912647ca9880b8a62aaf698b (127.0.103.2). New cstate: current_term: 1 leader_uuid: "e0b024e7912647ca9880b8a62aaf698b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0b024e7912647ca9880b8a62aaf698b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34963 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:15.395442 440 catalog_manager.cc:5526] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad reported cstate change: term changed from 0 to 1, leader changed from <none> to 78973ac01961460faf125c074bf617ad (127.0.103.3). New cstate: current_term: 1 leader_uuid: "78973ac01961460faf125c074bf617ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "78973ac01961460faf125c074bf617ad" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43329 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:15.401615 925 tablet_bootstrap.cc:654] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:15.434252 925 tablet_bootstrap.cc:492] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef: No bootstrap required, opened a new log
I20250111 02:51:15.434415 925 ts_tablet_manager.cc:1397] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.002s
I20250111 02:51:15.435019 925 raft_consensus.cc:357] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } }
I20250111 02:51:15.435168 925 raft_consensus.cc:383] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:15.435247 925 raft_consensus.cc:738] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c218f3b887f745e087e658e039accbef, State: Initialized, Role: FOLLOWER
I20250111 02:51:15.435477 925 consensus_queue.cc:260] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } }
I20250111 02:51:15.435607 925 raft_consensus.cc:397] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:15.435701 925 raft_consensus.cc:491] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:15.435781 925 raft_consensus.cc:3054] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:15.437338 925 raft_consensus.cc:513] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } }
I20250111 02:51:15.437511 925 leader_election.cc:304] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c218f3b887f745e087e658e039accbef; no voters:
I20250111 02:51:15.437685 925 leader_election.cc:290] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:15.438095 925 ts_tablet_manager.cc:1428] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250111 02:51:15.438158 937 raft_consensus.cc:2798] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:15.438335 937 raft_consensus.cc:695] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 1 LEADER]: Becoming Leader. State: Replica: c218f3b887f745e087e658e039accbef, State: Running, Role: LEADER
I20250111 02:51:15.438557 937 consensus_queue.cc:237] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } }
I20250111 02:51:15.440703 441 catalog_manager.cc:5526] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef reported cstate change: term changed from 0 to 1, leader changed from <none> to c218f3b887f745e087e658e039accbef (127.0.103.1). New cstate: current_term: 1 leader_uuid: "c218f3b887f745e087e658e039accbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c218f3b887f745e087e658e039accbef" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42057 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:15.465567 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:51:15.475032 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:15.475421 412 tablet_replica.cc:331] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef: stopping tablet replica
I20250111 02:51:15.475773 412 raft_consensus.cc:2238] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:15.476022 412 raft_consensus.cc:2267] T 1f8fe93e7ff0468cb2dc34c3759302f8 P c218f3b887f745e087e658e039accbef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:15.477331 412 tablet_replica.cc:331] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef: stopping tablet replica
I20250111 02:51:15.477478 412 raft_consensus.cc:2238] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:15.477614 412 raft_consensus.cc:2267] T 42a9570933174bd58ed53f1ab5df1144 P c218f3b887f745e087e658e039accbef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:15.482255 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:51:15.486965 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:51:15.494663 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:15.494964 412 tablet_replica.cc:331] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b: stopping tablet replica
I20250111 02:51:15.495124 412 raft_consensus.cc:2238] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:15.495290 412 raft_consensus.cc:2267] T 8e77959392c045b987a1670857b13c0f P e0b024e7912647ca9880b8a62aaf698b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:15.498972 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:51:15.501133 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:51:15.508332 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:15.508708 412 tablet_replica.cc:331] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad: stopping tablet replica
I20250111 02:51:15.508903 412 raft_consensus.cc:2238] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:15.509058 412 raft_consensus.cc:2267] T db1bfd7d7e5a484b8b8aee0676d8a99b P 78973ac01961460faf125c074bf617ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:15.522912 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:51:15.525501 412 master.cc:537] Master@127.0.103.62:44145 shutting down...
I20250111 02:51:15.533648 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:15.534049 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:15.534163 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 869e46bd0ed549fe9f42900c453f4fdd: stopping tablet replica
I20250111 02:51:15.556208 412 master.cc:559] Master@127.0.103.62:44145 shutdown complete.
I20250111 02:51:15.560240 412 master.cc:537] Master@127.0.103.61:33967 shutting down...
I20250111 02:51:15.567154 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:15.567338 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:15.567422 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 9081218c6e0d483bbbec242612898c58: stopping tablet replica
I20250111 02:51:15.581038 412 master.cc:559] Master@127.0.103.61:33967 shutdown complete.
I20250111 02:51:15.585618 412 master.cc:537] Master@127.0.103.60:35137 shutting down...
I20250111 02:51:15.593691 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:15.593937 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:15.594069 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3bd6c433dc994753a1c43b0f71edc787: stopping tablet replica
I20250111 02:51:15.607334 412 master.cc:559] Master@127.0.103.60:35137 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3059 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20250111 02:51:15.619822 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:43881
I20250111 02:51:15.620246 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:15.623088 949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:15.623989 950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:15.624733 952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:15.625689 412 server_base.cc:1034] running on GCE node
I20250111 02:51:15.626004 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:15.626086 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:15.626142 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563875626125 us; error 0 us; skew 500 ppm
I20250111 02:51:15.626302 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:15.627509 412 webserver.cc:458] Webserver started at http://127.0.103.62:38473/ using document root <none> and password file <none>
I20250111 02:51:15.627717 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:15.627790 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:15.627925 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:15.628573 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "7aeac240d6bb495c8ea72a76f50945c1"
format_stamp: "Formatted at 2025-01-11 02:51:15 on dist-test-slave-4xrj"
I20250111 02:51:15.630995 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:51:15.632493 957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.632846 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:15.632937 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "7aeac240d6bb495c8ea72a76f50945c1"
format_stamp: "Formatted at 2025-01-11 02:51:15 on dist-test-slave-4xrj"
I20250111 02:51:15.633028 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:15.648042 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:15.648566 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:15.661463 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:43881
I20250111 02:51:15.661484 1008 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:43881 every 8 connection(s)
I20250111 02:51:15.662799 1009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:15.666297 1009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1: Bootstrap starting.
I20250111 02:51:15.667536 1009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:15.668994 1009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1: No bootstrap required, opened a new log
I20250111 02:51:15.669575 1009 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac240d6bb495c8ea72a76f50945c1" member_type: VOTER }
I20250111 02:51:15.669700 1009 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:15.669801 1009 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aeac240d6bb495c8ea72a76f50945c1, State: Initialized, Role: FOLLOWER
I20250111 02:51:15.670091 1009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac240d6bb495c8ea72a76f50945c1" member_type: VOTER }
I20250111 02:51:15.670224 1009 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:15.670315 1009 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:15.670409 1009 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:15.671976 1009 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac240d6bb495c8ea72a76f50945c1" member_type: VOTER }
I20250111 02:51:15.672181 1009 leader_election.cc:304] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aeac240d6bb495c8ea72a76f50945c1; no voters:
I20250111 02:51:15.672536 1009 leader_election.cc:290] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:15.672636 1012 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:15.673081 1012 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 1 LEADER]: Becoming Leader. State: Replica: 7aeac240d6bb495c8ea72a76f50945c1, State: Running, Role: LEADER
I20250111 02:51:15.673333 1012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac240d6bb495c8ea72a76f50945c1" member_type: VOTER }
I20250111 02:51:15.673379 1009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:15.674259 1013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aeac240d6bb495c8ea72a76f50945c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac240d6bb495c8ea72a76f50945c1" member_type: VOTER } }
I20250111 02:51:15.674332 1014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aeac240d6bb495c8ea72a76f50945c1. Latest consensus state: current_term: 1 leader_uuid: "7aeac240d6bb495c8ea72a76f50945c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeac240d6bb495c8ea72a76f50945c1" member_type: VOTER } }
I20250111 02:51:15.674487 1013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:15.674571 1014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:15.675549 1017 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:51:15.676945 1017 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:51:15.679628 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:51:15.679745 1017 catalog_manager.cc:1348] Generated new cluster ID: 42d7147ce74d4b978439814fcc776185
I20250111 02:51:15.679843 1017 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:51:15.684967 1017 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:51:15.685724 1017 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:51:15.691533 1017 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1: Generated new TSK 0
I20250111 02:51:15.691725 1017 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:51:15.695873 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:15.699416 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:15.700506 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:15.701264 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:15.702077 412 server_base.cc:1034] running on GCE node
I20250111 02:51:15.702334 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:15.702405 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:15.702483 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563875702445 us; error 0 us; skew 500 ppm
I20250111 02:51:15.702634 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:15.703811 412 webserver.cc:458] Webserver started at http://127.0.103.1:39149/ using document root <none> and password file <none>
I20250111 02:51:15.704015 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:15.704107 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:15.704246 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:15.704854 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "8472d23e5fab44e192ad35cfbbfab063"
format_stamp: "Formatted at 2025-01-11 02:51:15 on dist-test-slave-4xrj"
I20250111 02:51:15.707262 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250111 02:51:15.708706 1038 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.709092 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:15.709182 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "8472d23e5fab44e192ad35cfbbfab063"
format_stamp: "Formatted at 2025-01-11 02:51:15 on dist-test-slave-4xrj"
I20250111 02:51:15.709309 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:15.718748 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:15.719252 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:15.719758 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:15.720513 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:15.720573 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.720643 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:15.720690 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.734841 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:41977
I20250111 02:51:15.734874 1100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:41977 every 8 connection(s)
I20250111 02:51:15.736408 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:15.740262 1105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:15.740995 1106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:15.741763 1108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:15.742928 412 server_base.cc:1034] running on GCE node
I20250111 02:51:15.743196 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:15.743294 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:15.743369 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563875743339 us; error 0 us; skew 500 ppm
I20250111 02:51:15.743376 1101 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43881
I20250111 02:51:15.743546 1101 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:15.743553 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:15.743901 1101 heartbeater.cc:510] Master 127.0.103.62:43881 requested a full tablet report, sending...
I20250111 02:51:15.744803 974 ts_manager.cc:194] Registered new tserver with Master: 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977)
I20250111 02:51:15.744983 412 webserver.cc:458] Webserver started at http://127.0.103.2:41855/ using document root <none> and password file <none>
I20250111 02:51:15.745270 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:15.745373 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:15.745532 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:15.746292 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "ccb94b5a97f24a208859b4240e31eb5f"
format_stamp: "Formatted at 2025-01-11 02:51:15 on dist-test-slave-4xrj"
I20250111 02:51:15.746414 974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60376
I20250111 02:51:15.748997 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:51:15.750418 1113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.750741 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:15.750837 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "ccb94b5a97f24a208859b4240e31eb5f"
format_stamp: "Formatted at 2025-01-11 02:51:15 on dist-test-slave-4xrj"
I20250111 02:51:15.750943 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:15.764405 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:15.764874 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:15.765403 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:15.766187 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:15.766273 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.766360 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:15.766449 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.779511 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:44087
I20250111 02:51:15.779560 1175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:44087 every 8 connection(s)
I20250111 02:51:15.781689 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:15.785238 1179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:15.786355 1180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:15.787376 412 server_base.cc:1034] running on GCE node
W20250111 02:51:15.787915 1182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:15.788287 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:15.788367 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:15.788450 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563875788405 us; error 0 us; skew 500 ppm
I20250111 02:51:15.788599 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:15.789741 1176 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43881
I20250111 02:51:15.789937 1176 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:15.790328 1176 heartbeater.cc:510] Master 127.0.103.62:43881 requested a full tablet report, sending...
I20250111 02:51:15.790562 412 webserver.cc:458] Webserver started at http://127.0.103.3:36501/ using document root <none> and password file <none>
I20250111 02:51:15.790784 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:15.790886 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:15.791019 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:15.791188 974 ts_manager.cc:194] Registered new tserver with Master: ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:15.791662 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
format_stamp: "Formatted at 2025-01-11 02:51:15 on dist-test-slave-4xrj"
I20250111 02:51:15.791894 974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60388
I20250111 02:51:15.793900 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:51:15.795284 1187 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.795590 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:15.795704 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
format_stamp: "Formatted at 2025-01-11 02:51:15 on dist-test-slave-4xrj"
I20250111 02:51:15.795821 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:15.804885 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:15.805389 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:15.805927 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:15.806671 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:15.806738 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.806850 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:15.806906 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:15.821017 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:42305
I20250111 02:51:15.821046 1249 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:42305 every 8 connection(s)
I20250111 02:51:15.827850 1250 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43881
I20250111 02:51:15.827984 1250 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:15.828372 1250 heartbeater.cc:510] Master 127.0.103.62:43881 requested a full tablet report, sending...
I20250111 02:51:15.829144 974 ts_manager.cc:194] Registered new tserver with Master: 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:15.829835 974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60404
I20250111 02:51:15.830183 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00778577s
I20250111 02:51:16.747977 1101 heartbeater.cc:502] Master 127.0.103.62:43881 was elected leader, sending a full tablet report...
I20250111 02:51:16.796746 1176 heartbeater.cc:502] Master 127.0.103.62:43881 was elected leader, sending a full tablet report...
I20250111 02:51:16.831241 1250 heartbeater.cc:502] Master 127.0.103.62:43881 was elected leader, sending a full tablet report...
I20250111 02:51:16.858042 412 test_util.cc:274] Using random seed: 1224902779
I20250111 02:51:16.871788 974 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60410:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:51:16.872504 974 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:51:16.898079 1214 tablet_service.cc:1467] Processing CreateTablet for tablet de092f9aabe74ff6bdfe0b6d20d713a0 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250111 02:51:16.898154 1215 tablet_service.cc:1467] Processing CreateTablet for tablet fb42d9075657415aac44fd75f34f63dc (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250111 02:51:16.898684 1214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de092f9aabe74ff6bdfe0b6d20d713a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.898931 1215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb42d9075657415aac44fd75f34f63dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.899022 1213 tablet_service.cc:1467] Processing CreateTablet for tablet 55627f2216f848d9bfc9ce32305a7406 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250111 02:51:16.899526 1213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55627f2216f848d9bfc9ce32305a7406. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.902997 1141 tablet_service.cc:1467] Processing CreateTablet for tablet fb42d9075657415aac44fd75f34f63dc (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250111 02:51:16.903548 1141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb42d9075657415aac44fd75f34f63dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.903631 1139 tablet_service.cc:1467] Processing CreateTablet for tablet 55627f2216f848d9bfc9ce32305a7406 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250111 02:51:16.903904 1212 tablet_service.cc:1467] Processing CreateTablet for tablet 60a042b3934a437c8980455066cdafb7 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250111 02:51:16.904049 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55627f2216f848d9bfc9ce32305a7406. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.903049 1140 tablet_service.cc:1467] Processing CreateTablet for tablet de092f9aabe74ff6bdfe0b6d20d713a0 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250111 02:51:16.904827 1208 tablet_service.cc:1467] Processing CreateTablet for tablet 83b02d8b36c047eba550168ebbcf638d (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250111 02:51:16.905050 1140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de092f9aabe74ff6bdfe0b6d20d713a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.905578 1138 tablet_service.cc:1467] Processing CreateTablet for tablet 60a042b3934a437c8980455066cdafb7 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250111 02:51:16.906168 1212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a042b3934a437c8980455066cdafb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.906968 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a042b3934a437c8980455066cdafb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.908141 1208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b02d8b36c047eba550168ebbcf638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.908170 1135 tablet_service.cc:1467] Processing CreateTablet for tablet 538e3b5abb2b4157aae4e1efee46cb2b (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250111 02:51:16.908142 1136 tablet_service.cc:1467] Processing CreateTablet for tablet 05a6bd112ad9469fa6a487c28aaf076e (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250111 02:51:16.909801 1209 tablet_service.cc:1467] Processing CreateTablet for tablet 538e3b5abb2b4157aae4e1efee46cb2b (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250111 02:51:16.910959 1209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538e3b5abb2b4157aae4e1efee46cb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.911060 1135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538e3b5abb2b4157aae4e1efee46cb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.910274 1210 tablet_service.cc:1467] Processing CreateTablet for tablet 05a6bd112ad9469fa6a487c28aaf076e (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250111 02:51:16.913038 1211 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa7fd2e7b94afd92865820b13b2b90 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250111 02:51:16.914023 1210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05a6bd112ad9469fa6a487c28aaf076e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.914505 1137 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa7fd2e7b94afd92865820b13b2b90 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250111 02:51:16.914569 1066 tablet_service.cc:1467] Processing CreateTablet for tablet fb42d9075657415aac44fd75f34f63dc (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250111 02:51:16.915042 1066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb42d9075657415aac44fd75f34f63dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.913748 1134 tablet_service.cc:1467] Processing CreateTablet for tablet 83b02d8b36c047eba550168ebbcf638d (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250111 02:51:16.916664 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05a6bd112ad9469fa6a487c28aaf076e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.918475 1211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecaa7fd2e7b94afd92865820b13b2b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.919555 1062 tablet_service.cc:1467] Processing CreateTablet for tablet ecaa7fd2e7b94afd92865820b13b2b90 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250111 02:51:16.919981 1062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecaa7fd2e7b94afd92865820b13b2b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.924041 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecaa7fd2e7b94afd92865820b13b2b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.927691 1063 tablet_service.cc:1467] Processing CreateTablet for tablet 60a042b3934a437c8980455066cdafb7 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250111 02:51:16.929747 1134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b02d8b36c047eba550168ebbcf638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.929642 1065 tablet_service.cc:1467] Processing CreateTablet for tablet de092f9aabe74ff6bdfe0b6d20d713a0 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250111 02:51:16.932447 1065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de092f9aabe74ff6bdfe0b6d20d713a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.933454 1060 tablet_service.cc:1467] Processing CreateTablet for tablet 538e3b5abb2b4157aae4e1efee46cb2b (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250111 02:51:16.935182 1064 tablet_service.cc:1467] Processing CreateTablet for tablet 55627f2216f848d9bfc9ce32305a7406 (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250111 02:51:16.935250 1061 tablet_service.cc:1467] Processing CreateTablet for tablet 05a6bd112ad9469fa6a487c28aaf076e (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250111 02:51:16.937413 1059 tablet_service.cc:1467] Processing CreateTablet for tablet 83b02d8b36c047eba550168ebbcf638d (DEFAULT_TABLE table=test-workload [id=4de84baeef544a5e902e66d7fe34cb95]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250111 02:51:16.941265 1063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a042b3934a437c8980455066cdafb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.942045 1271 tablet_bootstrap.cc:492] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f: Bootstrap starting.
I20250111 02:51:16.947228 1060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 538e3b5abb2b4157aae4e1efee46cb2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.951464 1061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05a6bd112ad9469fa6a487c28aaf076e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.958110 1064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55627f2216f848d9bfc9ce32305a7406. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.958451 1270 tablet_bootstrap.cc:492] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8: Bootstrap starting.
I20250111 02:51:16.959523 1059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 83b02d8b36c047eba550168ebbcf638d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:16.964206 1272 tablet_bootstrap.cc:492] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063: Bootstrap starting.
I20250111 02:51:16.970228 1272 tablet_bootstrap.cc:654] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:16.970283 1271 tablet_bootstrap.cc:654] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:16.985677 1270 tablet_bootstrap.cc:654] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:16.991798 1272 tablet_bootstrap.cc:492] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063: No bootstrap required, opened a new log
I20250111 02:51:16.992154 1272 ts_tablet_manager.cc:1397] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.009s
I20250111 02:51:16.994175 1272 raft_consensus.cc:357] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:16.994427 1272 raft_consensus.cc:383] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:16.994565 1272 raft_consensus.cc:738] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Initialized, Role: FOLLOWER
I20250111 02:51:16.994596 1270 tablet_bootstrap.cc:492] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8: No bootstrap required, opened a new log
I20250111 02:51:16.994743 1270 ts_tablet_manager.cc:1397] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.003s
I20250111 02:51:16.994810 1272 consensus_queue.cc:260] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:16.994997 1271 tablet_bootstrap.cc:492] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f: No bootstrap required, opened a new log
I20250111 02:51:16.995132 1271 ts_tablet_manager.cc:1397] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.001s
I20250111 02:51:16.995623 1270 raft_consensus.cc:357] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:16.995816 1270 raft_consensus.cc:383] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:16.995821 1272 ts_tablet_manager.cc:1428] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250111 02:51:16.995913 1270 raft_consensus.cc:738] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Initialized, Role: FOLLOWER
I20250111 02:51:16.996004 1271 raft_consensus.cc:357] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:16.996129 1272 tablet_bootstrap.cc:492] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063: Bootstrap starting.
I20250111 02:51:16.996207 1271 raft_consensus.cc:383] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:16.996296 1271 raft_consensus.cc:738] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Initialized, Role: FOLLOWER
I20250111 02:51:16.996181 1270 consensus_queue.cc:260] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:16.996515 1271 consensus_queue.cc:260] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:16.998260 1272 tablet_bootstrap.cc:654] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.002175 1272 tablet_bootstrap.cc:492] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063: No bootstrap required, opened a new log
I20250111 02:51:17.002336 1272 ts_tablet_manager.cc:1397] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20250111 02:51:17.002343 1271 ts_tablet_manager.cc:1428] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20250111 02:51:17.002619 1271 tablet_bootstrap.cc:492] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f: Bootstrap starting.
I20250111 02:51:17.003027 1272 raft_consensus.cc:357] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.003240 1272 raft_consensus.cc:383] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.003329 1272 raft_consensus.cc:738] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.003667 1272 consensus_queue.cc:260] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.004168 1272 ts_tablet_manager.cc:1428] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:51:17.004413 1271 tablet_bootstrap.cc:654] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.005002 1272 tablet_bootstrap.cc:492] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063: Bootstrap starting.
I20250111 02:51:17.007712 1272 tablet_bootstrap.cc:654] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.007772 1270 ts_tablet_manager.cc:1428] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20250111 02:51:17.008051 1270 tablet_bootstrap.cc:492] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8: Bootstrap starting.
I20250111 02:51:17.009891 1270 tablet_bootstrap.cc:654] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.016927 1271 tablet_bootstrap.cc:492] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f: No bootstrap required, opened a new log
I20250111 02:51:17.017164 1271 ts_tablet_manager.cc:1397] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.005s
I20250111 02:51:17.017911 1270 tablet_bootstrap.cc:492] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8: No bootstrap required, opened a new log
I20250111 02:51:17.018080 1270 ts_tablet_manager.cc:1397] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20250111 02:51:17.018541 1272 tablet_bootstrap.cc:492] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063: No bootstrap required, opened a new log
I20250111 02:51:17.018703 1272 ts_tablet_manager.cc:1397] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20250111 02:51:17.018709 1270 raft_consensus.cc:357] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.018894 1271 raft_consensus.cc:357] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.019027 1270 raft_consensus.cc:383] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.019096 1271 raft_consensus.cc:383] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.019129 1270 raft_consensus.cc:738] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.019186 1271 raft_consensus.cc:738] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.019476 1270 consensus_queue.cc:260] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.019451 1271 consensus_queue.cc:260] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.019536 1272 raft_consensus.cc:357] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.019798 1272 raft_consensus.cc:383] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.019898 1272 raft_consensus.cc:738] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.019996 1270 ts_tablet_manager.cc:1428] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:51:17.020128 1272 consensus_queue.cc:260] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.020313 1270 tablet_bootstrap.cc:492] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8: Bootstrap starting.
I20250111 02:51:17.020673 1271 ts_tablet_manager.cc:1428] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250111 02:51:17.021981 1271 tablet_bootstrap.cc:492] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f: Bootstrap starting.
I20250111 02:51:17.022888 1270 tablet_bootstrap.cc:654] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.023697 1271 tablet_bootstrap.cc:654] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.024376 1272 ts_tablet_manager.cc:1428] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250111 02:51:17.024668 1272 tablet_bootstrap.cc:492] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063: Bootstrap starting.
I20250111 02:51:17.026309 1272 tablet_bootstrap.cc:654] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.026372 1270 tablet_bootstrap.cc:492] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8: No bootstrap required, opened a new log
I20250111 02:51:17.026566 1270 ts_tablet_manager.cc:1397] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250111 02:51:17.027858 1270 raft_consensus.cc:357] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.028136 1270 raft_consensus.cc:383] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.028255 1270 raft_consensus.cc:738] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.028496 1272 tablet_bootstrap.cc:492] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063: No bootstrap required, opened a new log
I20250111 02:51:17.028657 1272 ts_tablet_manager.cc:1397] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:51:17.028589 1270 consensus_queue.cc:260] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.029207 1270 ts_tablet_manager.cc:1428] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20250111 02:51:17.029255 1272 raft_consensus.cc:357] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.029489 1272 raft_consensus.cc:383] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.029515 1270 tablet_bootstrap.cc:492] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8: Bootstrap starting.
I20250111 02:51:17.029575 1272 raft_consensus.cc:738] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.029839 1272 consensus_queue.cc:260] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.030478 1272 ts_tablet_manager.cc:1428] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:51:17.030778 1272 tablet_bootstrap.cc:492] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063: Bootstrap starting.
I20250111 02:51:17.031867 1270 tablet_bootstrap.cc:654] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.032505 1272 tablet_bootstrap.cc:654] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.034433 1271 tablet_bootstrap.cc:492] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f: No bootstrap required, opened a new log
I20250111 02:51:17.034602 1271 ts_tablet_manager.cc:1397] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20250111 02:51:17.035475 1271 raft_consensus.cc:357] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.035727 1271 raft_consensus.cc:383] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.035846 1271 raft_consensus.cc:738] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.036223 1271 consensus_queue.cc:260] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.036600 1272 tablet_bootstrap.cc:492] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063: No bootstrap required, opened a new log
I20250111 02:51:17.036744 1272 ts_tablet_manager.cc:1397] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20250111 02:51:17.037266 1272 raft_consensus.cc:357] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.037456 1272 raft_consensus.cc:383] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.037549 1272 raft_consensus.cc:738] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.037731 1271 ts_tablet_manager.cc:1428] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250111 02:51:17.037767 1272 consensus_queue.cc:260] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.038062 1271 tablet_bootstrap.cc:492] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f: Bootstrap starting.
I20250111 02:51:17.039006 1276 raft_consensus.cc:491] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:17.039615 1276 raft_consensus.cc:513] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.040621 1271 tablet_bootstrap.cc:654] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.040621 1270 tablet_bootstrap.cc:492] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8: No bootstrap required, opened a new log
I20250111 02:51:17.040681 1276 leader_election.cc:290] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.040827 1270 ts_tablet_manager.cc:1397] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250111 02:51:17.041512 1270 raft_consensus.cc:357] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.042395 1270 raft_consensus.cc:383] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.042460 1271 tablet_bootstrap.cc:492] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f: No bootstrap required, opened a new log
I20250111 02:51:17.042495 1270 raft_consensus.cc:738] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.042613 1271 ts_tablet_manager.cc:1397] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250111 02:51:17.042737 1270 consensus_queue.cc:260] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.043247 1270 ts_tablet_manager.cc:1428] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250111 02:51:17.043532 1270 tablet_bootstrap.cc:492] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8: Bootstrap starting.
I20250111 02:51:17.045274 1270 tablet_bootstrap.cc:654] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.047379 1272 ts_tablet_manager.cc:1428] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250111 02:51:17.047854 1271 raft_consensus.cc:357] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.048136 1271 raft_consensus.cc:383] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.048254 1271 raft_consensus.cc:738] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.048589 1271 consensus_queue.cc:260] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.049404 1271 ts_tablet_manager.cc:1428] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250111 02:51:17.049724 1271 tablet_bootstrap.cc:492] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f: Bootstrap starting.
I20250111 02:51:17.049955 1272 tablet_bootstrap.cc:492] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063: Bootstrap starting.
I20250111 02:51:17.051805 1272 tablet_bootstrap.cc:654] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.052008 1271 tablet_bootstrap.cc:654] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.053956 1270 tablet_bootstrap.cc:492] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8: No bootstrap required, opened a new log
I20250111 02:51:17.054132 1270 ts_tablet_manager.cc:1397] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20250111 02:51:17.055042 1270 raft_consensus.cc:357] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.055264 1270 raft_consensus.cc:383] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.055369 1270 raft_consensus.cc:738] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.055670 1270 consensus_queue.cc:260] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.057086 1270 ts_tablet_manager.cc:1428] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:51:17.057423 1270 tablet_bootstrap.cc:492] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8: Bootstrap starting.
I20250111 02:51:17.060421 1270 tablet_bootstrap.cc:654] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.062772 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb42d9075657415aac44fd75f34f63dc" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" is_pre_election: true
I20250111 02:51:17.063153 1225 raft_consensus.cc:2463] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 0.
I20250111 02:51:17.063761 1272 tablet_bootstrap.cc:492] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063: No bootstrap required, opened a new log
I20250111 02:51:17.063699 1040 leader_election.cc:304] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, 8472d23e5fab44e192ad35cfbbfab063; no voters:
I20250111 02:51:17.063974 1276 raft_consensus.cc:2798] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:17.064064 1276 raft_consensus.cc:491] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:17.064157 1276 raft_consensus.cc:3054] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.064515 1271 tablet_bootstrap.cc:492] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f: No bootstrap required, opened a new log
I20250111 02:51:17.064697 1271 ts_tablet_manager.cc:1397] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250111 02:51:17.065577 1271 raft_consensus.cc:357] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.065789 1271 raft_consensus.cc:383] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.065912 1271 raft_consensus.cc:738] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.066159 1270 tablet_bootstrap.cc:492] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8: No bootstrap required, opened a new log
I20250111 02:51:17.066151 1271 consensus_queue.cc:260] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.066352 1270 ts_tablet_manager.cc:1397] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20250111 02:51:17.066502 1276 raft_consensus.cc:513] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.066785 1271 ts_tablet_manager.cc:1428] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:51:17.067044 1271 tablet_bootstrap.cc:492] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f: Bootstrap starting.
I20250111 02:51:17.067009 1270 raft_consensus.cc:357] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.067793 1276 leader_election.cc:290] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 election: Requested vote from peers 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.067833 1270 raft_consensus.cc:383] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.067929 1270 raft_consensus.cc:738] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.067791 1272 ts_tablet_manager.cc:1397] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20250111 02:51:17.068086 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb42d9075657415aac44fd75f34f63dc" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
I20250111 02:51:17.068399 1225 raft_consensus.cc:3054] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.069039 1272 raft_consensus.cc:357] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.069590 1272 raft_consensus.cc:383] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.068400 1270 consensus_queue.cc:260] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.070138 1272 raft_consensus.cc:738] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.070724 1272 consensus_queue.cc:260] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.071244 1270 ts_tablet_manager.cc:1428] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250111 02:51:17.072984 1270 tablet_bootstrap.cc:492] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8: Bootstrap starting.
I20250111 02:51:17.073462 1271 tablet_bootstrap.cc:654] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.073802 1272 ts_tablet_manager.cc:1428] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250111 02:51:17.074218 1225 raft_consensus.cc:2463] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 1.
I20250111 02:51:17.074839 1272 tablet_bootstrap.cc:492] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063: Bootstrap starting.
I20250111 02:51:17.074870 1040 leader_election.cc:304] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, 8472d23e5fab44e192ad35cfbbfab063; no voters:
I20250111 02:51:17.076221 1276 raft_consensus.cc:2798] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:17.077967 1276 raft_consensus.cc:695] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Becoming Leader. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Running, Role: LEADER
I20250111 02:51:17.078449 1271 tablet_bootstrap.cc:492] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f: No bootstrap required, opened a new log
I20250111 02:51:17.078286 1276 consensus_queue.cc:237] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.078584 1270 tablet_bootstrap.cc:654] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.078632 1271 ts_tablet_manager.cc:1397] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20250111 02:51:17.079226 1271 raft_consensus.cc:357] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.079425 1271 raft_consensus.cc:383] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.076562 1272 tablet_bootstrap.cc:654] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.079528 1271 raft_consensus.cc:738] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.079891 1271 consensus_queue.cc:260] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.080417 1271 ts_tablet_manager.cc:1428] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250111 02:51:17.080705 1271 tablet_bootstrap.cc:492] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f: Bootstrap starting.
I20250111 02:51:17.081652 974 catalog_manager.cc:5526] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "8472d23e5fab44e192ad35cfbbfab063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:17.082346 1271 tablet_bootstrap.cc:654] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.085575 1270 tablet_bootstrap.cc:492] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8: No bootstrap required, opened a new log
I20250111 02:51:17.085821 1270 ts_tablet_manager.cc:1397] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20250111 02:51:17.087558 1270 raft_consensus.cc:357] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.087893 1270 raft_consensus.cc:383] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.088054 1270 raft_consensus.cc:738] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.088459 1270 consensus_queue.cc:260] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.089259 1270 ts_tablet_manager.cc:1428] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250111 02:51:17.089593 1270 tablet_bootstrap.cc:492] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8: Bootstrap starting.
I20250111 02:51:17.091498 1270 tablet_bootstrap.cc:654] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.092590 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb42d9075657415aac44fd75f34f63dc" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f" is_pre_election: true
I20250111 02:51:17.092907 1272 tablet_bootstrap.cc:492] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063: No bootstrap required, opened a new log
I20250111 02:51:17.092901 1151 raft_consensus.cc:2463] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 0.
I20250111 02:51:17.093070 1272 ts_tablet_manager.cc:1397] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.009s
I20250111 02:51:17.093825 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb42d9075657415aac44fd75f34f63dc" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:17.094184 1150 raft_consensus.cc:3054] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.094972 1271 tablet_bootstrap.cc:492] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f: No bootstrap required, opened a new log
I20250111 02:51:17.094969 1270 tablet_bootstrap.cc:492] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8: No bootstrap required, opened a new log
I20250111 02:51:17.095149 1271 ts_tablet_manager.cc:1397] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20250111 02:51:17.095515 1270 ts_tablet_manager.cc:1397] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250111 02:51:17.096736 1150 raft_consensus.cc:2463] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 1.
I20250111 02:51:17.097729 1271 raft_consensus.cc:357] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.097689 1272 raft_consensus.cc:357] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.097915 1271 raft_consensus.cc:383] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.097689 1270 raft_consensus.cc:357] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.098001 1272 raft_consensus.cc:383] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.098024 1271 raft_consensus.cc:738] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.098104 1270 raft_consensus.cc:383] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.098642 1272 raft_consensus.cc:738] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.098645 1270 raft_consensus.cc:738] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.098649 1271 consensus_queue.cc:260] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.098906 1272 consensus_queue.cc:260] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.098989 1270 consensus_queue.cc:260] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.099303 1271 ts_tablet_manager.cc:1428] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250111 02:51:17.099571 1271 tablet_bootstrap.cc:492] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f: Bootstrap starting.
I20250111 02:51:17.100026 1270 ts_tablet_manager.cc:1428] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250111 02:51:17.101300 1272 ts_tablet_manager.cc:1428] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250111 02:51:17.101330 1271 tablet_bootstrap.cc:654] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.101588 1272 tablet_bootstrap.cc:492] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063: Bootstrap starting.
I20250111 02:51:17.103560 1272 tablet_bootstrap.cc:654] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:17.105486 1272 tablet_bootstrap.cc:492] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063: No bootstrap required, opened a new log
I20250111 02:51:17.105664 1272 ts_tablet_manager.cc:1397] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:51:17.106299 1271 tablet_bootstrap.cc:492] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f: No bootstrap required, opened a new log
I20250111 02:51:17.106475 1271 ts_tablet_manager.cc:1397] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250111 02:51:17.106518 1272 raft_consensus.cc:357] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.106710 1272 raft_consensus.cc:383] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.106792 1272 raft_consensus.cc:738] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.107017 1272 consensus_queue.cc:260] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.107122 1271 raft_consensus.cc:357] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.107313 1271 raft_consensus.cc:383] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:17.107390 1271 raft_consensus.cc:738] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Initialized, Role: FOLLOWER
I20250111 02:51:17.107483 1272 ts_tablet_manager.cc:1428] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250111 02:51:17.107625 1271 consensus_queue.cc:260] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.108008 1271 ts_tablet_manager.cc:1428] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:51:17.118317 1277 raft_consensus.cc:491] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:17.118466 1277 raft_consensus.cc:513] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.119246 1277 leader_election.cc:290] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.125813 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a042b3934a437c8980455066cdafb7" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f" is_pre_election: true
I20250111 02:51:17.125876 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a042b3934a437c8980455066cdafb7" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8472d23e5fab44e192ad35cfbbfab063" is_pre_election: true
I20250111 02:51:17.126101 1150 raft_consensus.cc:2463] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 0.
I20250111 02:51:17.126159 1076 raft_consensus.cc:2463] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 0.
I20250111 02:51:17.126559 1189 leader_election.cc:304] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.126827 1277 raft_consensus.cc:2798] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:17.126943 1277 raft_consensus.cc:491] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:17.127035 1277 raft_consensus.cc:3054] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.128862 1277 raft_consensus.cc:513] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.129441 1277 leader_election.cc:290] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 election: Requested vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.129909 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a042b3934a437c8980455066cdafb7" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:17.129914 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60a042b3934a437c8980455066cdafb7" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
I20250111 02:51:17.130134 1076 raft_consensus.cc:3054] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.130146 1150 raft_consensus.cc:3054] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.132927 1150 raft_consensus.cc:2463] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 1.
I20250111 02:51:17.132928 1076 raft_consensus.cc:2463] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 1.
I20250111 02:51:17.133376 1190 leader_election.cc:304] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.133716 1277 raft_consensus.cc:2798] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:17.134132 1277 raft_consensus.cc:695] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Becoming Leader. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Running, Role: LEADER
I20250111 02:51:17.134439 1277 consensus_queue.cc:237] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.136917 974 catalog_manager.cc:5526] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:17.143513 1276 raft_consensus.cc:491] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:17.143668 1276 raft_consensus.cc:513] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.144191 1276 leader_election.cc:290] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.144629 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a6bd112ad9469fa6a487c28aaf076e" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f" is_pre_election: true
I20250111 02:51:17.144621 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a6bd112ad9469fa6a487c28aaf076e" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" is_pre_election: true
I20250111 02:51:17.144869 1150 raft_consensus.cc:2463] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 0.
I20250111 02:51:17.144876 1225 raft_consensus.cc:2463] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 0.
I20250111 02:51:17.145294 1276 raft_consensus.cc:491] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:17.145329 1040 leader_election.cc:304] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.145406 1276 raft_consensus.cc:513] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.145646 1296 raft_consensus.cc:2798] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:17.145745 1296 raft_consensus.cc:491] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:17.145892 1296 raft_consensus.cc:3054] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.146080 1276 leader_election.cc:290] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.146422 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538e3b5abb2b4157aae4e1efee46cb2b" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f" is_pre_election: true
I20250111 02:51:17.146497 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538e3b5abb2b4157aae4e1efee46cb2b" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" is_pre_election: true
I20250111 02:51:17.146624 1150 raft_consensus.cc:2463] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 0.
I20250111 02:51:17.147173 1225 raft_consensus.cc:2463] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 0.
I20250111 02:51:17.147181 1040 leader_election.cc:304] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.147552 1276 raft_consensus.cc:2798] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:17.147665 1276 raft_consensus.cc:491] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:17.147751 1276 raft_consensus.cc:3054] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.148943 1296 raft_consensus.cc:513] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.149505 1296 leader_election.cc:290] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 election: Requested vote from peers ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.149573 1276 raft_consensus.cc:513] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.149912 1225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a6bd112ad9469fa6a487c28aaf076e" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
I20250111 02:51:17.149914 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a6bd112ad9469fa6a487c28aaf076e" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:17.150158 1276 leader_election.cc:290] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 election: Requested vote from peers ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.150141 1150 raft_consensus.cc:3054] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.150148 1225 raft_consensus.cc:3054] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.150640 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538e3b5abb2b4157aae4e1efee46cb2b" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:17.150638 1224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538e3b5abb2b4157aae4e1efee46cb2b" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
I20250111 02:51:17.150923 1151 raft_consensus.cc:3054] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.150954 1224 raft_consensus.cc:3054] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.153887 1150 raft_consensus.cc:2463] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 1.
I20250111 02:51:17.154104 1225 raft_consensus.cc:2463] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 1.
I20250111 02:51:17.154304 1040 leader_election.cc:304] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.154312 1151 raft_consensus.cc:2463] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 1.
I20250111 02:51:17.154580 1276 raft_consensus.cc:2798] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:17.154727 1276 raft_consensus.cc:695] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Becoming Leader. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Running, Role: LEADER
I20250111 02:51:17.154759 1040 leader_election.cc:304] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.155045 1276 consensus_queue.cc:237] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.155536 1224 raft_consensus.cc:2463] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 1.
I20250111 02:51:17.156237 1296 raft_consensus.cc:2798] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:17.156407 1296 raft_consensus.cc:695] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Becoming Leader. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Running, Role: LEADER
I20250111 02:51:17.156579 1296 consensus_queue.cc:237] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.159883 974 catalog_manager.cc:5526] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "8472d23e5fab44e192ad35cfbbfab063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:17.163604 974 catalog_manager.cc:5526] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "8472d23e5fab44e192ad35cfbbfab063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:17.255218 1278 raft_consensus.cc:491] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:17.255435 1278 raft_consensus.cc:513] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.256301 1278 leader_election.cc:290] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.263010 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55627f2216f848d9bfc9ce32305a7406" candidate_uuid: "ccb94b5a97f24a208859b4240e31eb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8472d23e5fab44e192ad35cfbbfab063" is_pre_election: true
I20250111 02:51:17.263258 1076 raft_consensus.cc:2463] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb94b5a97f24a208859b4240e31eb5f in term 0.
I20250111 02:51:17.263262 1224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55627f2216f848d9bfc9ce32305a7406" candidate_uuid: "ccb94b5a97f24a208859b4240e31eb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" is_pre_election: true
I20250111 02:51:17.263531 1224 raft_consensus.cc:2463] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb94b5a97f24a208859b4240e31eb5f in term 0.
I20250111 02:51:17.263640 1116 leader_election.cc:304] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.263886 1278 raft_consensus.cc:2798] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:17.263988 1278 raft_consensus.cc:491] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:17.264077 1278 raft_consensus.cc:3054] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.265892 1278 raft_consensus.cc:513] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.266428 1278 leader_election.cc:290] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [CANDIDATE]: Term 1 election: Requested vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.266798 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55627f2216f848d9bfc9ce32305a7406" candidate_uuid: "ccb94b5a97f24a208859b4240e31eb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
I20250111 02:51:17.266845 1224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55627f2216f848d9bfc9ce32305a7406" candidate_uuid: "ccb94b5a97f24a208859b4240e31eb5f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
I20250111 02:51:17.266971 1076 raft_consensus.cc:3054] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.267010 1224 raft_consensus.cc:3054] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.268786 1224 raft_consensus.cc:2463] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb94b5a97f24a208859b4240e31eb5f in term 1.
I20250111 02:51:17.268782 1076 raft_consensus.cc:2463] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb94b5a97f24a208859b4240e31eb5f in term 1.
I20250111 02:51:17.269179 1115 leader_election.cc:304] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.269491 1278 raft_consensus.cc:2798] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:17.269879 1278 raft_consensus.cc:695] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 1 LEADER]: Becoming Leader. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Running, Role: LEADER
I20250111 02:51:17.270174 1278 consensus_queue.cc:237] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.272506 974 catalog_manager.cc:5526] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f reported cstate change: term changed from 0 to 1, leader changed from <none> to ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2). New cstate: current_term: 1 leader_uuid: "ccb94b5a97f24a208859b4240e31eb5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:17.284103 1277 raft_consensus.cc:491] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:17.284263 1277 raft_consensus.cc:513] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.284763 1277 leader_election.cc:290] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.285174 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b02d8b36c047eba550168ebbcf638d" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f" is_pre_election: true
I20250111 02:51:17.285176 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b02d8b36c047eba550168ebbcf638d" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8472d23e5fab44e192ad35cfbbfab063" is_pre_election: true
I20250111 02:51:17.285420 1076 raft_consensus.cc:2463] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 0.
I20250111 02:51:17.285427 1151 raft_consensus.cc:2463] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 0.
I20250111 02:51:17.285746 1190 leader_election.cc:304] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, 8472d23e5fab44e192ad35cfbbfab063; no voters:
I20250111 02:51:17.286038 1277 raft_consensus.cc:2798] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:17.286120 1277 raft_consensus.cc:491] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:17.286212 1277 raft_consensus.cc:3054] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.287873 1277 raft_consensus.cc:513] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.288388 1277 leader_election.cc:290] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 election: Requested vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.289038 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b02d8b36c047eba550168ebbcf638d" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
I20250111 02:51:17.289225 1076 raft_consensus.cc:3054] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.289203 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b02d8b36c047eba550168ebbcf638d" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:17.289460 1151 raft_consensus.cc:3054] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.291957 1076 raft_consensus.cc:2463] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 1.
I20250111 02:51:17.291952 1151 raft_consensus.cc:2463] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 1.
I20250111 02:51:17.292358 1189 leader_election.cc:304] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.292626 1277 raft_consensus.cc:2798] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:17.292804 1277 raft_consensus.cc:695] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Becoming Leader. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Running, Role: LEADER
I20250111 02:51:17.293061 1277 consensus_queue.cc:237] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.295387 974 catalog_manager.cc:5526] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:17.310886 1277 raft_consensus.cc:491] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:17.311112 1277 raft_consensus.cc:513] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.311700 1277 leader_election.cc:290] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.312189 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de092f9aabe74ff6bdfe0b6d20d713a0" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8472d23e5fab44e192ad35cfbbfab063" is_pre_election: true
I20250111 02:51:17.312217 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de092f9aabe74ff6bdfe0b6d20d713a0" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f" is_pre_election: true
I20250111 02:51:17.312453 1076 raft_consensus.cc:2463] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 0.
I20250111 02:51:17.312450 1151 raft_consensus.cc:2463] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 0.
I20250111 02:51:17.312687 1276 raft_consensus.cc:491] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:17.312865 1189 leader_election.cc:304] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.312896 1276 raft_consensus.cc:513] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.313119 1277 raft_consensus.cc:2798] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:17.313251 1277 raft_consensus.cc:491] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:17.313354 1277 raft_consensus.cc:3054] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.313503 1276 leader_election.cc:290] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.313879 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa7fd2e7b94afd92865820b13b2b90" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f" is_pre_election: true
I20250111 02:51:17.313941 1224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa7fd2e7b94afd92865820b13b2b90" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" is_pre_election: true
I20250111 02:51:17.314061 1151 raft_consensus.cc:2463] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 0.
I20250111 02:51:17.314160 1224 raft_consensus.cc:2463] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 0.
I20250111 02:51:17.314412 1040 leader_election.cc:304] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.314673 1276 raft_consensus.cc:2798] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:17.314767 1276 raft_consensus.cc:491] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:17.314872 1276 raft_consensus.cc:3054] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.315691 1277 raft_consensus.cc:513] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.316227 1277 leader_election.cc:290] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 election: Requested vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.316493 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de092f9aabe74ff6bdfe0b6d20d713a0" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
I20250111 02:51:17.316561 1276 raft_consensus.cc:513] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.316704 1076 raft_consensus.cc:3054] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.316681 1151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de092f9aabe74ff6bdfe0b6d20d713a0" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:17.316859 1151 raft_consensus.cc:3054] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.317907 1224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa7fd2e7b94afd92865820b13b2b90" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
I20250111 02:51:17.318022 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa7fd2e7b94afd92865820b13b2b90" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:17.318105 1224 raft_consensus.cc:3054] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.318253 1150 raft_consensus.cc:3054] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:17.320103 1076 raft_consensus.cc:2463] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 1.
I20250111 02:51:17.320104 1151 raft_consensus.cc:2463] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 1.
I20250111 02:51:17.320456 1189 leader_election.cc:304] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.320787 1277 raft_consensus.cc:2798] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:17.320933 1277 raft_consensus.cc:695] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Becoming Leader. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Running, Role: LEADER
I20250111 02:51:17.320966 1150 raft_consensus.cc:2463] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 1.
I20250111 02:51:17.321198 1277 consensus_queue.cc:237] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.321347 1040 leader_election.cc:304] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.317133 1276 leader_election.cc:290] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 1 election: Requested vote from peers ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.321620 1224 raft_consensus.cc:2463] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 1.
I20250111 02:51:17.321650 1296 raft_consensus.cc:2798] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:17.321784 1296 raft_consensus.cc:695] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Becoming Leader. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Running, Role: LEADER
I20250111 02:51:17.322067 1296 consensus_queue.cc:237] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.324045 974 catalog_manager.cc:5526] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:17.324512 972 catalog_manager.cc:5526] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "8472d23e5fab44e192ad35cfbbfab063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:17.372187 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:17.375761 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:17.376420 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:17.377121 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:17.377961 412 server_base.cc:1034] running on GCE node
I20250111 02:51:17.378254 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:17.378340 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:17.378404 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563877378386 us; error 0 us; skew 500 ppm
I20250111 02:51:17.378578 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:17.379922 412 webserver.cc:458] Webserver started at http://127.0.103.4:38745/ using document root <none> and password file <none>
I20250111 02:51:17.380159 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:17.380249 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:17.380401 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:17.381073 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-3-root/instance:
uuid: "1e759796f2ba4f4495a56a2423aa4c7c"
format_stamp: "Formatted at 2025-01-11 02:51:17 on dist-test-slave-4xrj"
I20250111 02:51:17.383560 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:51:17.385135 1326 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:17.385495 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250111 02:51:17.385601 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-3-root
uuid: "1e759796f2ba4f4495a56a2423aa4c7c"
format_stamp: "Formatted at 2025-01-11 02:51:17 on dist-test-slave-4xrj"
I20250111 02:51:17.385728 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1736563872491422-412-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:17.401573 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:17.402199 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:17.402724 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:17.403502 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:17.403580 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:17.403676 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:17.403756 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:17.420866 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.4:41149
I20250111 02:51:17.420904 1388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.4:41149 every 8 connection(s)
I20250111 02:51:17.428396 1389 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43881
I20250111 02:51:17.428536 1389 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:17.428933 1389 heartbeater.cc:510] Master 127.0.103.62:43881 requested a full tablet report, sending...
I20250111 02:51:17.429761 972 ts_manager.cc:194] Registered new tserver with Master: 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4:41149)
I20250111 02:51:17.430518 972 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60418
I20250111 02:51:17.537271 1277 consensus_queue.cc:1035] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.541797 1306 consensus_queue.cc:1035] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.571694 1296 consensus_queue.cc:1035] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.575737 1276 consensus_queue.cc:1035] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.597499 1276 consensus_queue.cc:1035] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.601373 1296 consensus_queue.cc:1035] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.679286 1027 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:17.687242 1224 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de092f9aabe74ff6bdfe0b6d20d713a0"
dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
mode: GRACEFUL
new_leader_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
from {username='slave'} at 127.0.0.1:39850
I20250111 02:51:17.687531 1224 raft_consensus.cc:604] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Received request to transfer leadership to TS ccb94b5a97f24a208859b4240e31eb5f
I20250111 02:51:17.693068 1076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ecaa7fd2e7b94afd92865820b13b2b90"
dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
mode: GRACEFUL
new_leader_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
from {username='slave'} at 127.0.0.1:52288
I20250111 02:51:17.693241 1076 raft_consensus.cc:604] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Received request to transfer leadership to TS ccb94b5a97f24a208859b4240e31eb5f
I20250111 02:51:17.694252 1076 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb42d9075657415aac44fd75f34f63dc"
dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
mode: GRACEFUL
new_leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
from {username='slave'} at 127.0.0.1:52288
I20250111 02:51:17.694386 1076 raft_consensus.cc:604] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Received request to transfer leadership to TS 14b6cd8d671a44b68444e9efed3f3cb8
I20250111 02:51:17.694756 1027 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250111 02:51:17.695025 1027 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:17.734503 1278 consensus_queue.cc:1035] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.738396 1276 consensus_queue.cc:1035] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.738775 1405 consensus_queue.cc:1035] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.744036 1308 consensus_queue.cc:1035] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.778064 1308 consensus_queue.cc:1035] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.778084 1395 consensus_queue.cc:1035] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.782948 1395 consensus_queue.cc:1035] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.784859 1276 consensus_queue.cc:1035] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.787917 1296 raft_consensus.cc:988] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063: : Instructing follower ccb94b5a97f24a208859b4240e31eb5f to start an election
I20250111 02:51:17.788097 1296 raft_consensus.cc:1076] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Signalling peer ccb94b5a97f24a208859b4240e31eb5f to start an election
I20250111 02:51:17.789096 1151 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ecaa7fd2e7b94afd92865820b13b2b90"
dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
from {username='slave'} at 127.0.0.1:40268
I20250111 02:51:17.789348 1151 raft_consensus.cc:491] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:51:17.789511 1151 raft_consensus.cc:3054] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.791636 1151 raft_consensus.cc:513] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.792227 1151 leader_election.cc:290] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [CANDIDATE]: Term 2 election: Requested vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.792600 1076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa7fd2e7b94afd92865820b13b2b90" candidate_uuid: "ccb94b5a97f24a208859b4240e31eb5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
I20250111 02:51:17.792639 1224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecaa7fd2e7b94afd92865820b13b2b90" candidate_uuid: "ccb94b5a97f24a208859b4240e31eb5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
I20250111 02:51:17.792789 1076 raft_consensus.cc:3049] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:51:17.792820 1224 raft_consensus.cc:3054] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.792912 1076 raft_consensus.cc:738] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Running, Role: LEADER
I20250111 02:51:17.793190 1076 consensus_queue.cc:260] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.793576 1076 raft_consensus.cc:3054] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.795266 1224 raft_consensus.cc:2463] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb94b5a97f24a208859b4240e31eb5f in term 2.
I20250111 02:51:17.795680 1115 leader_election.cc:304] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.795974 1278 raft_consensus.cc:2798] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:51:17.796100 1278 raft_consensus.cc:695] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 2 LEADER]: Becoming Leader. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Running, Role: LEADER
I20250111 02:51:17.796183 1076 raft_consensus.cc:2463] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb94b5a97f24a208859b4240e31eb5f in term 2.
I20250111 02:51:17.796352 1278 consensus_queue.cc:237] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.798568 973 catalog_manager.cc:5526] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f reported cstate change: term changed from 1 to 2, leader changed from 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1) to ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2). New cstate: current_term: 2 leader_uuid: "ccb94b5a97f24a208859b4240e31eb5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:17.941188 1395 consensus_queue.cc:1035] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.943552 1277 raft_consensus.cc:988] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8: : Instructing follower ccb94b5a97f24a208859b4240e31eb5f to start an election
I20250111 02:51:17.943722 1277 raft_consensus.cc:1076] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Signalling peer ccb94b5a97f24a208859b4240e31eb5f to start an election
I20250111 02:51:17.944525 1150 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de092f9aabe74ff6bdfe0b6d20d713a0"
dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
from {username='slave'} at 127.0.0.1:40280
I20250111 02:51:17.944775 1150 raft_consensus.cc:491] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:51:17.944912 1150 raft_consensus.cc:3054] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.946213 1306 consensus_queue.cc:1035] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:17.947646 1150 raft_consensus.cc:513] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.948428 1150 leader_election.cc:290] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [CANDIDATE]: Term 2 election: Requested vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:17.948736 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de092f9aabe74ff6bdfe0b6d20d713a0" candidate_uuid: "ccb94b5a97f24a208859b4240e31eb5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
I20250111 02:51:17.948987 1075 raft_consensus.cc:3054] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.949038 1224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de092f9aabe74ff6bdfe0b6d20d713a0" candidate_uuid: "ccb94b5a97f24a208859b4240e31eb5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
I20250111 02:51:17.949251 1224 raft_consensus.cc:3049] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:51:17.949380 1224 raft_consensus.cc:738] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Running, Role: LEADER
I20250111 02:51:17.949771 1224 consensus_queue.cc:260] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.950214 1224 raft_consensus.cc:3054] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.951525 1075 raft_consensus.cc:2463] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb94b5a97f24a208859b4240e31eb5f in term 2.
I20250111 02:51:17.952411 1116 leader_election.cc:304] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.952672 1278 raft_consensus.cc:2798] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:51:17.952845 1278 raft_consensus.cc:695] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 2 LEADER]: Becoming Leader. State: Replica: ccb94b5a97f24a208859b4240e31eb5f, State: Running, Role: LEADER
I20250111 02:51:17.953073 1224 raft_consensus.cc:2463] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccb94b5a97f24a208859b4240e31eb5f in term 2.
I20250111 02:51:17.953125 1278 consensus_queue.cc:237] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:17.955195 974 catalog_manager.cc:5526] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f reported cstate change: term changed from 1 to 2, leader changed from 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3) to ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2). New cstate: current_term: 2 leader_uuid: "ccb94b5a97f24a208859b4240e31eb5f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:17.967684 1308 raft_consensus.cc:988] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063: : Instructing follower 14b6cd8d671a44b68444e9efed3f3cb8 to start an election
I20250111 02:51:17.967823 1308 raft_consensus.cc:1076] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Signalling peer 14b6cd8d671a44b68444e9efed3f3cb8 to start an election
I20250111 02:51:17.968407 1224 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb42d9075657415aac44fd75f34f63dc"
dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
from {username='slave'} at 127.0.0.1:39828
I20250111 02:51:17.968585 1224 raft_consensus.cc:491] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:51:17.968712 1224 raft_consensus.cc:3054] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.970517 1224 raft_consensus.cc:513] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.971086 1224 leader_election.cc:290] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 2 election: Requested vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:17.971457 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb42d9075657415aac44fd75f34f63dc" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
I20250111 02:51:17.971457 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb42d9075657415aac44fd75f34f63dc" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:17.971688 1075 raft_consensus.cc:3049] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:51:17.971771 1150 raft_consensus.cc:3054] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.971776 1075 raft_consensus.cc:738] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Running, Role: LEADER
I20250111 02:51:17.972059 1075 consensus_queue.cc:260] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.972348 1075 raft_consensus.cc:3054] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:17.973675 1150 raft_consensus.cc:2463] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 2.
I20250111 02:51:17.974043 1189 leader_election.cc:304] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:17.974126 1075 raft_consensus.cc:2463] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 2.
I20250111 02:51:17.974340 1306 raft_consensus.cc:2798] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:51:17.974506 1306 raft_consensus.cc:695] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 LEADER]: Becoming Leader. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Running, Role: LEADER
I20250111 02:51:17.974817 1306 consensus_queue.cc:237] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } }
I20250111 02:51:17.976668 974 catalog_manager.cc:5526] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 reported cstate change: term changed from 1 to 2, leader changed from 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1) to 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3). New cstate: current_term: 2 leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:18.346572 1224 raft_consensus.cc:1270] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Refusing update from remote peer ccb94b5a97f24a208859b4240e31eb5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:18.347048 1405 consensus_queue.cc:1035] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:18.349942 1075 raft_consensus.cc:1270] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Refusing update from remote peer ccb94b5a97f24a208859b4240e31eb5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:18.350968 1405 consensus_queue.cc:1035] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:18.356503 1075 raft_consensus.cc:1270] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Refusing update from remote peer ccb94b5a97f24a208859b4240e31eb5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:18.356935 1405 consensus_queue.cc:1035] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:18.359822 1224 raft_consensus.cc:1270] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Refusing update from remote peer ccb94b5a97f24a208859b4240e31eb5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:18.360689 1405 consensus_queue.cc:1035] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:18.431597 1389 heartbeater.cc:502] Master 127.0.103.62:43881 was elected leader, sending a full tablet report...
I20250111 02:51:18.502590 1075 raft_consensus.cc:1270] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Refusing update from remote peer 14b6cd8d671a44b68444e9efed3f3cb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:18.503101 1306 consensus_queue.cc:1035] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:18.505780 1150 raft_consensus.cc:1270] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Refusing update from remote peer 14b6cd8d671a44b68444e9efed3f3cb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:18.506953 1395 consensus_queue.cc:1035] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:19.695443 1027 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:19.697614 1224 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83b02d8b36c047eba550168ebbcf638d"
dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
mode: GRACEFUL
new_leader_uuid: "8472d23e5fab44e192ad35cfbbfab063"
from {username='slave'} at 127.0.0.1:39850
I20250111 02:51:19.697799 1224 raft_consensus.cc:604] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Received request to transfer leadership to TS 8472d23e5fab44e192ad35cfbbfab063
I20250111 02:51:19.698309 1027 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:51:19.698500 1027 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:19.720064 1395 raft_consensus.cc:988] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8: : Instructing follower 8472d23e5fab44e192ad35cfbbfab063 to start an election
I20250111 02:51:19.720204 1395 raft_consensus.cc:1076] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Signalling peer 8472d23e5fab44e192ad35cfbbfab063 to start an election
I20250111 02:51:19.720852 1075 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83b02d8b36c047eba550168ebbcf638d"
dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
from {username='slave'} at 127.0.0.1:52268
I20250111 02:51:19.721024 1075 raft_consensus.cc:491] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:51:19.721122 1075 raft_consensus.cc:3054] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:19.722836 1075 raft_consensus.cc:513] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:19.723351 1075 leader_election.cc:290] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 2 election: Requested vote from peers ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087), 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3:42305)
I20250111 02:51:19.723706 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b02d8b36c047eba550168ebbcf638d" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:19.723812 1224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b02d8b36c047eba550168ebbcf638d" candidate_uuid: "8472d23e5fab44e192ad35cfbbfab063" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
I20250111 02:51:19.723930 1150 raft_consensus.cc:3054] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:19.724033 1224 raft_consensus.cc:3049] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:51:19.724130 1224 raft_consensus.cc:738] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Running, Role: LEADER
I20250111 02:51:19.724426 1224 consensus_queue.cc:260] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:19.724715 1224 raft_consensus.cc:3054] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:19.725932 1150 raft_consensus.cc:2463] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 2.
I20250111 02:51:19.726409 1040 leader_election.cc:304] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8472d23e5fab44e192ad35cfbbfab063, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:19.726557 1224 raft_consensus.cc:2463] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8472d23e5fab44e192ad35cfbbfab063 in term 2.
I20250111 02:51:19.726651 1296 raft_consensus.cc:2798] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:51:19.727308 1296 raft_consensus.cc:695] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 2 LEADER]: Becoming Leader. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Running, Role: LEADER
I20250111 02:51:19.727591 1296 consensus_queue.cc:237] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:19.730013 974 catalog_manager.cc:5526] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 reported cstate change: term changed from 1 to 2, leader changed from 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3) to 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1). New cstate: current_term: 2 leader_uuid: "8472d23e5fab44e192ad35cfbbfab063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:20.118484 1224 raft_consensus.cc:1270] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Refusing update from remote peer 8472d23e5fab44e192ad35cfbbfab063: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:20.119042 1296 consensus_queue.cc:1035] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:20.122835 1150 raft_consensus.cc:1270] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Refusing update from remote peer 8472d23e5fab44e192ad35cfbbfab063: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:20.123328 1296 consensus_queue.cc:1035] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:21.698894 1027 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:21.701040 1075 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "538e3b5abb2b4157aae4e1efee46cb2b"
dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
mode: GRACEFUL
new_leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
from {username='slave'} at 127.0.0.1:52288
I20250111 02:51:21.701256 1075 raft_consensus.cc:604] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Received request to transfer leadership to TS 14b6cd8d671a44b68444e9efed3f3cb8
I20250111 02:51:21.701673 1027 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:51:21.701818 1027 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:21.956269 1428 raft_consensus.cc:988] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063: : Instructing follower 14b6cd8d671a44b68444e9efed3f3cb8 to start an election
I20250111 02:51:21.956408 1428 raft_consensus.cc:1076] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Signalling peer 14b6cd8d671a44b68444e9efed3f3cb8 to start an election
I20250111 02:51:21.956945 1225 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "538e3b5abb2b4157aae4e1efee46cb2b"
dest_uuid: "14b6cd8d671a44b68444e9efed3f3cb8"
from {username='slave'} at 127.0.0.1:39828
I20250111 02:51:21.957110 1225 raft_consensus.cc:491] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:51:21.957242 1225 raft_consensus.cc:3054] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:21.958945 1225 raft_consensus.cc:513] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:21.959509 1225 leader_election.cc:290] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 2 election: Requested vote from peers 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1:41977), ccb94b5a97f24a208859b4240e31eb5f (127.0.103.2:44087)
I20250111 02:51:21.959934 1150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538e3b5abb2b4157aae4e1efee46cb2b" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ccb94b5a97f24a208859b4240e31eb5f"
I20250111 02:51:21.960031 1075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "538e3b5abb2b4157aae4e1efee46cb2b" candidate_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8472d23e5fab44e192ad35cfbbfab063"
I20250111 02:51:21.960170 1150 raft_consensus.cc:3054] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:21.960191 1075 raft_consensus.cc:3049] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:51:21.960292 1075 raft_consensus.cc:738] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8472d23e5fab44e192ad35cfbbfab063, State: Running, Role: LEADER
I20250111 02:51:21.960556 1075 consensus_queue.cc:260] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:21.960958 1075 raft_consensus.cc:3054] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:21.962622 1150 raft_consensus.cc:2463] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 2.
I20250111 02:51:21.962988 1189 leader_election.cc:304] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b6cd8d671a44b68444e9efed3f3cb8, ccb94b5a97f24a208859b4240e31eb5f; no voters:
I20250111 02:51:21.963191 1075 raft_consensus.cc:2463] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b6cd8d671a44b68444e9efed3f3cb8 in term 2.
I20250111 02:51:21.963282 1395 raft_consensus.cc:2798] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:51:21.963709 1395 raft_consensus.cc:695] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 LEADER]: Becoming Leader. State: Replica: 14b6cd8d671a44b68444e9efed3f3cb8, State: Running, Role: LEADER
I20250111 02:51:21.963958 1395 consensus_queue.cc:237] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } }
I20250111 02:51:21.966048 974 catalog_manager.cc:5526] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 reported cstate change: term changed from 1 to 2, leader changed from 8472d23e5fab44e192ad35cfbbfab063 (127.0.103.1) to 14b6cd8d671a44b68444e9efed3f3cb8 (127.0.103.3). New cstate: current_term: 2 leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:22.368562 1150 raft_consensus.cc:1270] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Refusing update from remote peer 14b6cd8d671a44b68444e9efed3f3cb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:22.369144 1395 consensus_queue.cc:1035] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:22.372879 1075 raft_consensus.cc:1270] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Refusing update from remote peer 14b6cd8d671a44b68444e9efed3f3cb8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:51:22.373407 1395 consensus_queue.cc:1035] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:22.688129 1225 consensus_queue.cc:237] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } }
I20250111 02:51:22.690223 1150 raft_consensus.cc:1270] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Refusing update from remote peer 14b6cd8d671a44b68444e9efed3f3cb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:51:22.690371 1075 raft_consensus.cc:1270] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Refusing update from remote peer 14b6cd8d671a44b68444e9efed3f3cb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:51:22.690897 1443 consensus_queue.cc:1035] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:51:22.692425 1443 consensus_queue.cc:1035] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:51:22.693511 1443 raft_consensus.cc:2949] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.694643 1151 raft_consensus.cc:2949] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.696557 1075 raft_consensus.cc:2949] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.700703 973 catalog_manager.cc:5526] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 reported cstate change: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New cstate: current_term: 2 leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250111 02:51:22.702078 1191 consensus_peers.cc:487] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 -> Peer 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4:41149): Couldn't send request to peer 1e759796f2ba4f4495a56a2423aa4c7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 538e3b5abb2b4157aae4e1efee46cb2b. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:51:22.707916 1151 consensus_queue.cc:237] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } attrs { replace: true } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } }
I20250111 02:51:22.710523 1075 raft_consensus.cc:1270] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Refusing update from remote peer ccb94b5a97f24a208859b4240e31eb5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:51:22.711050 1451 consensus_queue.cc:1035] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:51:22.711287 1225 raft_consensus.cc:1270] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Refusing update from remote peer ccb94b5a97f24a208859b4240e31eb5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:51:22.711875 1451 consensus_queue.cc:1035] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:51:22.713609 1278 raft_consensus.cc:2949] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } attrs { replace: true } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.714118 1075 raft_consensus.cc:2949] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } attrs { replace: true } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
W20250111 02:51:22.717664 1117 consensus_peers.cc:487] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f -> Peer 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4:41149): Couldn't send request to peer 1e759796f2ba4f4495a56a2423aa4c7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ecaa7fd2e7b94afd92865820b13b2b90. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:51:22.719055 972 catalog_manager.cc:5526] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 reported cstate change: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New cstate: current_term: 2 leader_uuid: "ccb94b5a97f24a208859b4240e31eb5f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } attrs { replace: true } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.722244 1225 raft_consensus.cc:2949] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } attrs { replace: true } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.728860 1075 consensus_queue.cc:237] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } }
I20250111 02:51:22.731487 1151 raft_consensus.cc:1270] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Refusing update from remote peer 8472d23e5fab44e192ad35cfbbfab063: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:51:22.731555 1225 raft_consensus.cc:1270] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Refusing update from remote peer 8472d23e5fab44e192ad35cfbbfab063: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:51:22.731954 1428 consensus_queue.cc:1035] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:22.732259 1460 consensus_queue.cc:1035] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:22.734607 1460 raft_consensus.cc:2949] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.735112 1151 raft_consensus.cc:2949] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
W20250111 02:51:22.739528 1042 consensus_peers.cc:487] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 -> Peer 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4:41149): Couldn't send request to peer 1e759796f2ba4f4495a56a2423aa4c7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05a6bd112ad9469fa6a487c28aaf076e. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:51:22.740159 1225 raft_consensus.cc:2949] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.740597 974 catalog_manager.cc:5526] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f reported cstate change: config changed from index -1 to 2, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "8472d23e5fab44e192ad35cfbbfab063" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.743126 1224 consensus_queue.cc:237] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } }
I20250111 02:51:22.745357 1075 raft_consensus.cc:1270] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Refusing update from remote peer 14b6cd8d671a44b68444e9efed3f3cb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:51:22.745718 1151 raft_consensus.cc:1270] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Refusing update from remote peer 14b6cd8d671a44b68444e9efed3f3cb8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:51:22.746068 1443 consensus_queue.cc:1035] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:51:22.746316 1430 consensus_queue.cc:1035] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250111 02:51:22.746604 1191 consensus_peers.cc:487] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 -> Peer 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4:41149): Couldn't send request to peer 1e759796f2ba4f4495a56a2423aa4c7c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb42d9075657415aac44fd75f34f63dc. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:51:22.749030 1430 raft_consensus.cc:2949] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.750433 1151 raft_consensus.cc:2949] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.751725 1075 raft_consensus.cc:2949] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } } }
I20250111 02:51:22.753690 973 catalog_manager.cc:5526] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 reported cstate change: config changed from index -1 to 3, NON_VOTER 1e759796f2ba4f4495a56a2423aa4c7c (127.0.103.4) added. New cstate: current_term: 2 leader_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8472d23e5fab44e192ad35cfbbfab063" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 41977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14b6cd8d671a44b68444e9efed3f3cb8" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 42305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ccb94b5a97f24a208859b4240e31eb5f" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 44087 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e759796f2ba4f4495a56a2423aa4c7c" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 41149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:22.935554 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
W20250111 02:51:22.942715 960 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
I20250111 02:51:22.943648 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:22.943930 412 tablet_replica.cc:331] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063: stopping tablet replica
W20250111 02:51:22.944061 1022 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250111 02:51:22.944123 412 raft_consensus.cc:2238] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250111 02:51:22.944154 1022 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250111 02:51:22.944332 412 raft_consensus.cc:2267] T 55627f2216f848d9bfc9ce32305a7406 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.944973 412 tablet_replica.cc:331] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063: stopping tablet replica
I20250111 02:51:22.945166 412 raft_consensus.cc:2238] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.945397 412 raft_consensus.cc:2267] T 538e3b5abb2b4157aae4e1efee46cb2b P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.946059 412 tablet_replica.cc:331] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063: stopping tablet replica
I20250111 02:51:22.946281 412 raft_consensus.cc:2238] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:22.946677 412 raft_consensus.cc:2267] T 05a6bd112ad9469fa6a487c28aaf076e P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.947314 412 tablet_replica.cc:331] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063: stopping tablet replica
I20250111 02:51:22.947448 412 raft_consensus.cc:2238] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.947592 412 raft_consensus.cc:2267] T fb42d9075657415aac44fd75f34f63dc P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.948228 412 tablet_replica.cc:331] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063: stopping tablet replica
I20250111 02:51:22.948383 412 raft_consensus.cc:2238] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.948554 412 raft_consensus.cc:2267] T ecaa7fd2e7b94afd92865820b13b2b90 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.949172 412 tablet_replica.cc:331] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063: stopping tablet replica
I20250111 02:51:22.949328 412 raft_consensus.cc:2238] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:51:22.949565 412 raft_consensus.cc:2267] T 83b02d8b36c047eba550168ebbcf638d P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.950088 412 tablet_replica.cc:331] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063: stopping tablet replica
I20250111 02:51:22.950227 412 raft_consensus.cc:2238] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.950389 412 raft_consensus.cc:2267] T 60a042b3934a437c8980455066cdafb7 P 8472d23e5fab44e192ad35cfbbfab063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.950790 412 tablet_replica.cc:331] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063: stopping tablet replica
I20250111 02:51:22.950954 412 raft_consensus.cc:2238] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.951104 412 raft_consensus.cc:2267] T de092f9aabe74ff6bdfe0b6d20d713a0 P 8472d23e5fab44e192ad35cfbbfab063 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.967027 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:51:22.971050 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
W20250111 02:51:22.972524 1022 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250111 02:51:22.972635 1022 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250111 02:51:22.981065 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:22.981293 412 tablet_replica.cc:331] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f: stopping tablet replica
I20250111 02:51:22.981525 412 raft_consensus.cc:2238] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.981732 412 raft_consensus.cc:2267] T 05a6bd112ad9469fa6a487c28aaf076e P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.982556 412 tablet_replica.cc:331] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f: stopping tablet replica
I20250111 02:51:22.982729 412 raft_consensus.cc:2238] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:22.982997 412 raft_consensus.cc:2267] T 55627f2216f848d9bfc9ce32305a7406 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.983493 412 tablet_replica.cc:331] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f: stopping tablet replica
I20250111 02:51:22.983704 412 raft_consensus.cc:2238] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:51:22.983934 412 raft_consensus.cc:2267] T de092f9aabe74ff6bdfe0b6d20d713a0 P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.984431 412 tablet_replica.cc:331] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f: stopping tablet replica
I20250111 02:51:22.984593 412 raft_consensus.cc:2238] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:51:22.984937 412 raft_consensus.cc:2267] T ecaa7fd2e7b94afd92865820b13b2b90 P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.985548 412 tablet_replica.cc:331] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f: stopping tablet replica
I20250111 02:51:22.985708 412 raft_consensus.cc:2238] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.985904 412 raft_consensus.cc:2267] T fb42d9075657415aac44fd75f34f63dc P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.986534 412 tablet_replica.cc:331] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f: stopping tablet replica
I20250111 02:51:22.986665 412 raft_consensus.cc:2238] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.986836 412 raft_consensus.cc:2267] T 538e3b5abb2b4157aae4e1efee46cb2b P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.987399 412 tablet_replica.cc:331] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f: stopping tablet replica
I20250111 02:51:22.987550 412 raft_consensus.cc:2238] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.987685 412 raft_consensus.cc:2267] T 60a042b3934a437c8980455066cdafb7 P ccb94b5a97f24a208859b4240e31eb5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.988066 412 tablet_replica.cc:331] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f: stopping tablet replica
I20250111 02:51:22.988194 412 raft_consensus.cc:2238] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:22.988339 412 raft_consensus.cc:2267] T 83b02d8b36c047eba550168ebbcf638d P ccb94b5a97f24a208859b4240e31eb5f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:22.994153 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:51:22.998587 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
W20250111 02:51:23.000967 1022 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250111 02:51:23.001354 1022 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250111 02:51:23.003800 1022 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250111 02:51:23.003942 1022 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250111 02:51:23.007277 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:23.007535 412 tablet_replica.cc:331] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8: stopping tablet replica
I20250111 02:51:23.007688 412 raft_consensus.cc:2238] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:23.007874 412 raft_consensus.cc:2267] T 05a6bd112ad9469fa6a487c28aaf076e P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.008554 412 tablet_replica.cc:331] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8: stopping tablet replica
I20250111 02:51:23.008713 412 raft_consensus.cc:2238] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:51:23.009030 412 raft_consensus.cc:2267] T 538e3b5abb2b4157aae4e1efee46cb2b P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.009613 412 tablet_replica.cc:331] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8: stopping tablet replica
I20250111 02:51:23.009763 412 raft_consensus.cc:2238] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:23.009964 412 raft_consensus.cc:2267] T 55627f2216f848d9bfc9ce32305a7406 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.010351 412 tablet_replica.cc:331] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8: stopping tablet replica
I20250111 02:51:23.010491 412 raft_consensus.cc:2238] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:23.010627 412 raft_consensus.cc:2267] T de092f9aabe74ff6bdfe0b6d20d713a0 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.010985 412 tablet_replica.cc:331] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8: stopping tablet replica
I20250111 02:51:23.011108 412 raft_consensus.cc:2238] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:23.011281 412 raft_consensus.cc:2267] T ecaa7fd2e7b94afd92865820b13b2b90 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.011766 412 tablet_replica.cc:331] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8: stopping tablet replica
I20250111 02:51:23.011895 412 raft_consensus.cc:2238] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:51:23.012145 412 raft_consensus.cc:2267] T fb42d9075657415aac44fd75f34f63dc P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.012625 412 tablet_replica.cc:331] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8: stopping tablet replica
I20250111 02:51:23.012745 412 raft_consensus.cc:2238] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:23.012882 412 raft_consensus.cc:2267] T 83b02d8b36c047eba550168ebbcf638d P 14b6cd8d671a44b68444e9efed3f3cb8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.013231 412 tablet_replica.cc:331] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8: stopping tablet replica
I20250111 02:51:23.013358 412 raft_consensus.cc:2238] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:23.013556 412 raft_consensus.cc:2267] T 60a042b3934a437c8980455066cdafb7 P 14b6cd8d671a44b68444e9efed3f3cb8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.027920 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:51:23.031387 412 tablet_server.cc:178] TabletServer@127.0.103.4:0 shutting down...
I20250111 02:51:23.038523 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:23.050987 412 tablet_server.cc:195] TabletServer@127.0.103.4:0 shutdown complete.
I20250111 02:51:23.053356 412 master.cc:537] Master@127.0.103.62:43881 shutting down...
I20250111 02:51:23.059906 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:23.060125 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:23.060240 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7aeac240d6bb495c8ea72a76f50945c1: stopping tablet replica
I20250111 02:51:23.073503 412 master.cc:559] Master@127.0.103.62:43881 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7468 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20250111 02:51:23.088725 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:37347,127.0.103.61:44163,127.0.103.60:41545
I20250111 02:51:23.089305 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:23.092188 1469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.093031 1470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.094013 1472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.094856 412 server_base.cc:1034] running on GCE node
I20250111 02:51:23.095089 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:23.095155 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:23.095238 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563883095205 us; error 0 us; skew 500 ppm
I20250111 02:51:23.095391 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:23.096591 412 webserver.cc:458] Webserver started at http://127.0.103.62:41167/ using document root <none> and password file <none>
I20250111 02:51:23.096797 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:23.096892 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:23.097021 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:23.097615 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "12999f39682d496a8389b0b500d76d5a"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.099972 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250111 02:51:23.101502 1477 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.101830 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:51:23.101956 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "12999f39682d496a8389b0b500d76d5a"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.102116 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:23.112263 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:23.112740 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:23.124462 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:37347
I20250111 02:51:23.124492 1528 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:37347 every 8 connection(s)
I20250111 02:51:23.125694 1529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:23.125730 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:51:23.128638 1529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:51:23.128964 1531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.130038 1532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.130897 1534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.131723 412 server_base.cc:1034] running on GCE node
I20250111 02:51:23.131963 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:23.132038 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:23.132112 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563883132076 us; error 0 us; skew 500 ppm
I20250111 02:51:23.132270 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:23.137941 412 webserver.cc:458] Webserver started at http://127.0.103.61:39693/ using document root <none> and password file <none>
I20250111 02:51:23.138154 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:23.138268 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:23.138404 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:23.138723 1529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:23.139127 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root/instance:
uuid: "2332d77de4da4b28b16a9f5ed89c700d"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
W20250111 02:51:23.140529 1529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.61:44163: Network error: Client connection negotiation failed: client connection to 127.0.103.61:44163: connect: Connection refused (error 111)
I20250111 02:51:23.141752 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:51:23.143271 1542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.143591 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:23.143693 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root
uuid: "2332d77de4da4b28b16a9f5ed89c700d"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.143811 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:23.163105 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:23.163637 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:23.175272 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.61:44163
I20250111 02:51:23.175311 1593 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.61:44163 every 8 connection(s)
I20250111 02:51:23.176609 1594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:23.176614 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:23.179944 1597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.179986 1594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:51:23.180744 1596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.181399 1599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.182348 412 server_base.cc:1034] running on GCE node
I20250111 02:51:23.182608 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:23.182698 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:23.182763 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563883182733 us; error 0 us; skew 500 ppm
I20250111 02:51:23.182927 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:23.184234 412 webserver.cc:458] Webserver started at http://127.0.103.60:44545/ using document root <none> and password file <none>
I20250111 02:51:23.184505 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:23.184613 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:23.184814 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:23.185563 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root/instance:
uuid: "861c39487cb84dd8b88bebfc32b601c8"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.186261 1594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:23.188198 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250111 02:51:23.189991 1606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.190357 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250111 02:51:23.190490 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root
uuid: "861c39487cb84dd8b88bebfc32b601c8"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.190651 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:23.190734 1529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } attempt: 1
I20250111 02:51:23.197705 1594 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:51:23.200078 1594 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.60:41545: Network error: Client connection negotiation failed: client connection to 127.0.103.60:41545: connect: Connection refused (error 111)
I20250111 02:51:23.200438 1529 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } has no permanent_uuid. Determining permanent_uuid...
W20250111 02:51:23.201699 1529 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.60:41545: Network error: Client connection negotiation failed: client connection to 127.0.103.60:41545: connect: Connection refused (error 111)
I20250111 02:51:23.225018 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:23.225626 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:23.237983 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.60:41545
I20250111 02:51:23.238018 1659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.60:41545 every 8 connection(s)
I20250111 02:51:23.238972 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:51:23.239351 1660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:23.241376 1660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:23.243261 1594 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } attempt: 1
I20250111 02:51:23.247887 1660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:23.251794 1594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d: Bootstrap starting.
I20250111 02:51:23.253453 1594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:23.253466 1660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } has no permanent_uuid. Determining permanent_uuid...
I20250111 02:51:23.255343 1594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d: No bootstrap required, opened a new log
I20250111 02:51:23.255980 1594 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.256196 1594 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:23.256270 1594 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2332d77de4da4b28b16a9f5ed89c700d, State: Initialized, Role: FOLLOWER
I20250111 02:51:23.256549 1594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.256871 1529 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } attempt: 1
I20250111 02:51:23.257761 1594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:23.257766 1665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.258067 1665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:23.263099 1660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: Bootstrap starting.
I20250111 02:51:23.265479 1660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: Neither blocks nor log segments found. Creating new log.
W20250111 02:51:23.266145 1677 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:51:23.266294 1677 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:51:23.267042 1529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a: Bootstrap starting.
I20250111 02:51:23.267345 1660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: No bootstrap required, opened a new log
I20250111 02:51:23.267988 1660 raft_consensus.cc:357] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.268138 1660 raft_consensus.cc:383] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:23.268220 1660 raft_consensus.cc:738] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861c39487cb84dd8b88bebfc32b601c8, State: Initialized, Role: FOLLOWER
I20250111 02:51:23.268311 1529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:23.268482 1660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.269141 1679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.269418 1679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:23.269448 1660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:23.270011 1529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a: No bootstrap required, opened a new log
I20250111 02:51:23.270541 1529 raft_consensus.cc:357] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.270699 1529 raft_consensus.cc:383] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:23.270781 1529 raft_consensus.cc:738] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12999f39682d496a8389b0b500d76d5a, State: Initialized, Role: FOLLOWER
I20250111 02:51:23.271639 1529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.272409 1529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:23.272353 1685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.272686 1685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: This master's current role is: FOLLOWER
W20250111 02:51:23.277168 1697 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:51:23.277287 1697 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250111 02:51:23.277771 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250111 02:51:23.277889 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250111 02:51:23.278656 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:23.278890 1703 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250111 02:51:23.278983 1703 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250111 02:51:23.282104 1704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.282872 1705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.283536 1707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.284587 412 server_base.cc:1034] running on GCE node
I20250111 02:51:23.284790 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:23.284850 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:23.284894 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563883284882 us; error 0 us; skew 500 ppm
I20250111 02:51:23.285002 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:23.286252 412 webserver.cc:458] Webserver started at http://127.0.103.1:39485/ using document root <none> and password file <none>
I20250111 02:51:23.286461 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:23.286522 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:23.286635 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:23.287181 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "65e8e1d29a1447bfa6bae10cc7b98bc6"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.290040 412 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250111 02:51:23.291441 1712 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.291764 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:23.291864 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "65e8e1d29a1447bfa6bae10cc7b98bc6"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.291992 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:23.304869 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:23.305451 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:23.306036 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:23.307381 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:23.307469 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.307567 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:23.307636 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.312038 1665 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:23.312233 1665 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.313232 1665 leader_election.cc:290] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12999f39682d496a8389b0b500d76d5a (127.0.103.62:37347), 861c39487cb84dd8b88bebfc32b601c8 (127.0.103.60:41545)
I20250111 02:51:23.313650 1504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2332d77de4da4b28b16a9f5ed89c700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12999f39682d496a8389b0b500d76d5a" is_pre_election: true
I20250111 02:51:23.313884 1504 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2332d77de4da4b28b16a9f5ed89c700d in term 0.
I20250111 02:51:23.314023 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2332d77de4da4b28b16a9f5ed89c700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861c39487cb84dd8b88bebfc32b601c8" is_pre_election: true
I20250111 02:51:23.314319 1635 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2332d77de4da4b28b16a9f5ed89c700d in term 0.
I20250111 02:51:23.314426 1546 leader_election.cc:304] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12999f39682d496a8389b0b500d76d5a, 2332d77de4da4b28b16a9f5ed89c700d; no voters:
I20250111 02:51:23.314707 1665 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:23.314931 1665 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:23.315052 1665 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:23.317657 1665 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.318382 1665 leader_election.cc:290] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [CANDIDATE]: Term 1 election: Requested vote from peers 12999f39682d496a8389b0b500d76d5a (127.0.103.62:37347), 861c39487cb84dd8b88bebfc32b601c8 (127.0.103.60:41545)
I20250111 02:51:23.318810 1504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2332d77de4da4b28b16a9f5ed89c700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12999f39682d496a8389b0b500d76d5a"
I20250111 02:51:23.318915 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2332d77de4da4b28b16a9f5ed89c700d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "861c39487cb84dd8b88bebfc32b601c8"
I20250111 02:51:23.319031 1504 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:23.319157 1635 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:23.320824 1504 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2332d77de4da4b28b16a9f5ed89c700d in term 1.
I20250111 02:51:23.321153 1546 leader_election.cc:304] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12999f39682d496a8389b0b500d76d5a, 2332d77de4da4b28b16a9f5ed89c700d; no voters:
I20250111 02:51:23.321439 1665 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:23.321830 1665 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 1 LEADER]: Becoming Leader. State: Replica: 2332d77de4da4b28b16a9f5ed89c700d, State: Running, Role: LEADER
I20250111 02:51:23.322099 1665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:23.322654 1635 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2332d77de4da4b28b16a9f5ed89c700d in term 1.
I20250111 02:51:23.323400 1753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2332d77de4da4b28b16a9f5ed89c700d. Latest consensus state: current_term: 1 leader_uuid: "2332d77de4da4b28b16a9f5ed89c700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.323597 1753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:23.324174 1755 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:51:23.325578 1755 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:51:23.329519 1504 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 1 FOLLOWER]: Refusing update from remote peer 2332d77de4da4b28b16a9f5ed89c700d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:51:23.329520 1635 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 1 FOLLOWER]: Refusing update from remote peer 2332d77de4da4b28b16a9f5ed89c700d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:51:23.330119 1665 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:23.330307 1753 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:23.334228 1679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2332d77de4da4b28b16a9f5ed89c700d. Latest consensus state: current_term: 1 leader_uuid: "2332d77de4da4b28b16a9f5ed89c700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.334520 1679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:23.343212 1679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2332d77de4da4b28b16a9f5ed89c700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.343472 1679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:23.344820 1665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2332d77de4da4b28b16a9f5ed89c700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.345053 1665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:23.345217 1685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2332d77de4da4b28b16a9f5ed89c700d. Latest consensus state: current_term: 1 leader_uuid: "2332d77de4da4b28b16a9f5ed89c700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.345556 1685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:23.345467 1755 catalog_manager.cc:1348] Generated new cluster ID: 94244db43bb14435a62cf6e5d5b15395
I20250111 02:51:23.345759 1755 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:51:23.349053 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:33231
I20250111 02:51:23.349071 1784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:33231 every 8 connection(s)
I20250111 02:51:23.352911 1753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2332d77de4da4b28b16a9f5ed89c700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.353631 1753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:23.381711 1685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2332d77de4da4b28b16a9f5ed89c700d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:23.382026 1685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:23.388176 1786 heartbeater.cc:346] Connected to a master server at 127.0.103.62:37347
I20250111 02:51:23.388348 1786 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.389125 1786 heartbeater.cc:510] Master 127.0.103.62:37347 requested a full tablet report, sending...
I20250111 02:51:23.390533 1494 ts_manager.cc:194] Registered new tserver with Master: 65e8e1d29a1447bfa6bae10cc7b98bc6 (127.0.103.1:33231)
I20250111 02:51:23.391506 1785 heartbeater.cc:346] Connected to a master server at 127.0.103.60:41545
I20250111 02:51:23.391602 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:51:23.391631 1785 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.392076 1785 heartbeater.cc:510] Master 127.0.103.60:41545 requested a full tablet report, sending...
I20250111 02:51:23.393101 1625 ts_manager.cc:194] Registered new tserver with Master: 65e8e1d29a1447bfa6bae10cc7b98bc6 (127.0.103.1:33231)
W20250111 02:51:23.396951 1793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.397611 1794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.397818 1787 heartbeater.cc:346] Connected to a master server at 127.0.103.61:44163
I20250111 02:51:23.397950 1787 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.398386 1787 heartbeater.cc:510] Master 127.0.103.61:44163 requested a full tablet report, sending...
I20250111 02:51:23.399410 1559 ts_manager.cc:194] Registered new tserver with Master: 65e8e1d29a1447bfa6bae10cc7b98bc6 (127.0.103.1:33231)
W20250111 02:51:23.400101 1796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.401261 412 server_base.cc:1034] running on GCE node
I20250111 02:51:23.401527 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:23.401589 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:23.401636 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563883401609 us; error 0 us; skew 500 ppm
I20250111 02:51:23.401739 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:23.403375 1755 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:51:23.403904 412 webserver.cc:458] Webserver started at http://127.0.103.2:38259/ using document root <none> and password file <none>
I20250111 02:51:23.404219 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:23.404342 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:23.404398 1755 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:51:23.404505 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:23.405318 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "bd3c17ccd9984434b1ca5179fb168438"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.408463 412 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250111 02:51:23.410108 1801 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.410449 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250111 02:51:23.410566 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "bd3c17ccd9984434b1ca5179fb168438"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.410679 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:23.415920 1755 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d: Generated new TSK 0
I20250111 02:51:23.416144 1755 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:51:23.418862 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:23.419348 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:23.419867 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:23.420624 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:23.420686 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.420783 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:23.420832 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.436738 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:36717
I20250111 02:51:23.436767 1863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:36717 every 8 connection(s)
I20250111 02:51:23.448122 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:51:23.449599 1864 heartbeater.cc:346] Connected to a master server at 127.0.103.60:41545
I20250111 02:51:23.449738 1864 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.450275 1864 heartbeater.cc:510] Master 127.0.103.60:41545 requested a full tablet report, sending...
I20250111 02:51:23.451229 1625 ts_manager.cc:194] Registered new tserver with Master: bd3c17ccd9984434b1ca5179fb168438 (127.0.103.2:36717)
I20250111 02:51:23.457015 1865 heartbeater.cc:346] Connected to a master server at 127.0.103.62:37347
I20250111 02:51:23.457190 1865 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.457871 1865 heartbeater.cc:510] Master 127.0.103.62:37347 requested a full tablet report, sending...
W20250111 02:51:23.458698 1871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.459203 1494 ts_manager.cc:194] Registered new tserver with Master: bd3c17ccd9984434b1ca5179fb168438 (127.0.103.2:36717)
I20250111 02:51:23.459272 1866 heartbeater.cc:346] Connected to a master server at 127.0.103.61:44163
I20250111 02:51:23.459400 1866 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.459720 1866 heartbeater.cc:510] Master 127.0.103.61:44163 requested a full tablet report, sending...
I20250111 02:51:23.460431 1559 ts_manager.cc:194] Registered new tserver with Master: bd3c17ccd9984434b1ca5179fb168438 (127.0.103.2:36717)
I20250111 02:51:23.461411 1559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47972
W20250111 02:51:23.463996 1872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:23.464856 1874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:23.465688 412 server_base.cc:1034] running on GCE node
I20250111 02:51:23.465951 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:23.466023 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:23.466065 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563883466052 us; error 0 us; skew 500 ppm
I20250111 02:51:23.466188 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:23.467299 412 webserver.cc:458] Webserver started at http://127.0.103.3:33885/ using document root <none> and password file <none>
I20250111 02:51:23.467499 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:23.467561 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:23.467658 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:23.468152 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "ec851bd29c3140e7939076908d1fc8e5"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.470445 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:51:23.471771 1879 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.472070 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:51:23.472160 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "ec851bd29c3140e7939076908d1fc8e5"
format_stamp: "Formatted at 2025-01-11 02:51:23 on dist-test-slave-4xrj"
I20250111 02:51:23.472259 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:23.488898 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:23.489902 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:23.490438 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:23.491218 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:23.491297 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.491397 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:23.491468 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:23.506464 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:37417
I20250111 02:51:23.506522 1941 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:37417 every 8 connection(s)
I20250111 02:51:23.516670 1942 heartbeater.cc:346] Connected to a master server at 127.0.103.60:41545
I20250111 02:51:23.516808 1942 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.517174 1944 heartbeater.cc:346] Connected to a master server at 127.0.103.61:44163
I20250111 02:51:23.517361 1944 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.517336 1942 heartbeater.cc:510] Master 127.0.103.60:41545 requested a full tablet report, sending...
I20250111 02:51:23.517707 1944 heartbeater.cc:510] Master 127.0.103.61:44163 requested a full tablet report, sending...
I20250111 02:51:23.518325 1625 ts_manager.cc:194] Registered new tserver with Master: ec851bd29c3140e7939076908d1fc8e5 (127.0.103.3:37417)
I20250111 02:51:23.519133 1943 heartbeater.cc:346] Connected to a master server at 127.0.103.62:37347
I20250111 02:51:23.519184 1559 ts_manager.cc:194] Registered new tserver with Master: ec851bd29c3140e7939076908d1fc8e5 (127.0.103.3:37417)
I20250111 02:51:23.519253 1943 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:23.519637 1943 heartbeater.cc:510] Master 127.0.103.62:37347 requested a full tablet report, sending...
I20250111 02:51:23.520125 1559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47980
I20250111 02:51:23.520424 1494 ts_manager.cc:194] Registered new tserver with Master: ec851bd29c3140e7939076908d1fc8e5 (127.0.103.3:37417)
I20250111 02:51:23.520622 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011288925s
I20250111 02:51:24.278489 1697 catalog_manager.cc:1260] Loaded cluster ID: 94244db43bb14435a62cf6e5d5b15395
I20250111 02:51:24.278604 1697 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: loading cluster ID for follower catalog manager: success
I20250111 02:51:24.280058 1703 catalog_manager.cc:1260] Loaded cluster ID: 94244db43bb14435a62cf6e5d5b15395
I20250111 02:51:24.280138 1703 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a: loading cluster ID for follower catalog manager: success
I20250111 02:51:24.280639 1697 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: acquiring CA information for follower catalog manager: success
I20250111 02:51:24.281574 1697 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:51:24.281929 1703 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a: acquiring CA information for follower catalog manager: success
I20250111 02:51:24.282824 1703 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250111 02:51:24.401235 1559 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47962
I20250111 02:51:24.462473 1866 heartbeater.cc:502] Master 127.0.103.61:44163 was elected leader, sending a full tablet report...
I20250111 02:51:24.521273 1944 heartbeater.cc:502] Master 127.0.103.61:44163 was elected leader, sending a full tablet report...
I20250111 02:51:25.265008 1676 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:25.402323 1787 heartbeater.cc:502] Master 127.0.103.61:44163 was elected leader, sending a full tablet report...
I20250111 02:51:25.584249 412 test_util.cc:274] Using random seed: 1233629015
I20250111 02:51:25.622381 1559 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35748:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:51:25.662199 1829 tablet_service.cc:1467] Processing CreateTablet for tablet ccafb989d312485f91955d845e3e3920 (DEFAULT_TABLE table=test-workload [id=d304fce92fa04aac9c5a9bbf91940251]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250111 02:51:25.662947 1829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccafb989d312485f91955d845e3e3920. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:25.672439 1979 tablet_bootstrap.cc:492] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438: Bootstrap starting.
I20250111 02:51:25.674752 1979 tablet_bootstrap.cc:654] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:25.676901 1740 tablet_service.cc:1467] Processing CreateTablet for tablet 7731c2211e9e42b09d8737989e9ab575 (DEFAULT_TABLE table=test-workload [id=d304fce92fa04aac9c5a9bbf91940251]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250111 02:51:25.677361 1740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7731c2211e9e42b09d8737989e9ab575. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:25.677309 1907 tablet_service.cc:1467] Processing CreateTablet for tablet a7ef0cef0d5641268fe59dbd9449c8a4 (DEFAULT_TABLE table=test-workload [id=d304fce92fa04aac9c5a9bbf91940251]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250111 02:51:25.677763 1907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7ef0cef0d5641268fe59dbd9449c8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:25.681170 1979 tablet_bootstrap.cc:492] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438: No bootstrap required, opened a new log
I20250111 02:51:25.681393 1979 ts_tablet_manager.cc:1397] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20250111 02:51:25.682422 1979 raft_consensus.cc:357] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3c17ccd9984434b1ca5179fb168438" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 36717 } }
I20250111 02:51:25.682762 1979 raft_consensus.cc:383] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:25.682952 1979 raft_consensus.cc:738] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3c17ccd9984434b1ca5179fb168438, State: Initialized, Role: FOLLOWER
I20250111 02:51:25.683354 1979 consensus_queue.cc:260] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3c17ccd9984434b1ca5179fb168438" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 36717 } }
I20250111 02:51:25.683708 1979 raft_consensus.cc:397] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:25.683861 1979 raft_consensus.cc:491] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:25.684054 1979 raft_consensus.cc:3054] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:25.685410 1981 tablet_bootstrap.cc:492] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5: Bootstrap starting.
I20250111 02:51:25.686924 1982 tablet_bootstrap.cc:492] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6: Bootstrap starting.
I20250111 02:51:25.687592 1979 raft_consensus.cc:513] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3c17ccd9984434b1ca5179fb168438" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 36717 } }
I20250111 02:51:25.687836 1979 leader_election.cc:304] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd3c17ccd9984434b1ca5179fb168438; no voters:
I20250111 02:51:25.687969 1981 tablet_bootstrap.cc:654] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:25.688390 1979 leader_election.cc:290] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:25.688611 1983 raft_consensus.cc:2798] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:25.688998 1982 tablet_bootstrap.cc:654] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:25.692543 1983 raft_consensus.cc:695] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 1 LEADER]: Becoming Leader. State: Replica: bd3c17ccd9984434b1ca5179fb168438, State: Running, Role: LEADER
I20250111 02:51:25.692987 1983 consensus_queue.cc:237] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3c17ccd9984434b1ca5179fb168438" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 36717 } }
I20250111 02:51:25.702936 1559 catalog_manager.cc:5526] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd3c17ccd9984434b1ca5179fb168438 (127.0.103.2). New cstate: current_term: 1 leader_uuid: "bd3c17ccd9984434b1ca5179fb168438" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd3c17ccd9984434b1ca5179fb168438" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 36717 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:25.722575 1979 ts_tablet_manager.cc:1428] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438: Time spent starting tablet: real 0.041s user 0.016s sys 0.021s
I20250111 02:51:25.723798 1981 tablet_bootstrap.cc:492] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5: No bootstrap required, opened a new log
I20250111 02:51:25.724036 1981 ts_tablet_manager.cc:1397] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.027s
I20250111 02:51:25.724346 1982 tablet_bootstrap.cc:492] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6: No bootstrap required, opened a new log
I20250111 02:51:25.724474 1982 ts_tablet_manager.cc:1397] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.028s
I20250111 02:51:25.724849 1981 raft_consensus.cc:357] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec851bd29c3140e7939076908d1fc8e5" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37417 } }
I20250111 02:51:25.725020 1981 raft_consensus.cc:383] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:25.725145 1981 raft_consensus.cc:738] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec851bd29c3140e7939076908d1fc8e5, State: Initialized, Role: FOLLOWER
I20250111 02:51:25.725181 1982 raft_consensus.cc:357] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e8e1d29a1447bfa6bae10cc7b98bc6" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 33231 } }
I20250111 02:51:25.725541 1981 consensus_queue.cc:260] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec851bd29c3140e7939076908d1fc8e5" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37417 } }
I20250111 02:51:25.725744 1982 raft_consensus.cc:383] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:25.725883 1981 raft_consensus.cc:397] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:25.726035 1981 raft_consensus.cc:491] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:25.726217 1981 raft_consensus.cc:3054] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:25.725924 1982 raft_consensus.cc:738] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65e8e1d29a1447bfa6bae10cc7b98bc6, State: Initialized, Role: FOLLOWER
I20250111 02:51:25.727901 1982 consensus_queue.cc:260] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e8e1d29a1447bfa6bae10cc7b98bc6" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 33231 } }
I20250111 02:51:25.728509 1982 raft_consensus.cc:397] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:25.728655 1982 raft_consensus.cc:491] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:25.728824 1982 raft_consensus.cc:3054] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:25.729569 1981 raft_consensus.cc:513] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec851bd29c3140e7939076908d1fc8e5" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37417 } }
I20250111 02:51:25.729914 1981 leader_election.cc:304] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec851bd29c3140e7939076908d1fc8e5; no voters:
I20250111 02:51:25.730516 1981 leader_election.cc:290] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:25.730683 1990 raft_consensus.cc:2798] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:25.731931 1982 raft_consensus.cc:513] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e8e1d29a1447bfa6bae10cc7b98bc6" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 33231 } }
I20250111 02:51:25.732290 1982 leader_election.cc:304] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65e8e1d29a1447bfa6bae10cc7b98bc6; no voters:
I20250111 02:51:25.738992 1993 raft_consensus.cc:2798] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:25.739187 1982 leader_election.cc:290] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:25.740326 1990 raft_consensus.cc:695] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 1 LEADER]: Becoming Leader. State: Replica: ec851bd29c3140e7939076908d1fc8e5, State: Running, Role: LEADER
I20250111 02:51:25.740676 1990 consensus_queue.cc:237] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec851bd29c3140e7939076908d1fc8e5" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37417 } }
I20250111 02:51:25.743288 1981 ts_tablet_manager.cc:1428] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5: Time spent starting tablet: real 0.019s user 0.004s sys 0.014s
I20250111 02:51:25.748093 1993 raft_consensus.cc:695] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 1 LEADER]: Becoming Leader. State: Replica: 65e8e1d29a1447bfa6bae10cc7b98bc6, State: Running, Role: LEADER
I20250111 02:51:25.748503 1993 consensus_queue.cc:237] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e8e1d29a1447bfa6bae10cc7b98bc6" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 33231 } }
I20250111 02:51:25.748525 1982 ts_tablet_manager.cc:1428] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6: Time spent starting tablet: real 0.024s user 0.008s sys 0.008s
I20250111 02:51:25.750244 1559 catalog_manager.cc:5526] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec851bd29c3140e7939076908d1fc8e5 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "ec851bd29c3140e7939076908d1fc8e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec851bd29c3140e7939076908d1fc8e5" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37417 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:25.751145 1558 catalog_manager.cc:5526] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65e8e1d29a1447bfa6bae10cc7b98bc6 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "65e8e1d29a1447bfa6bae10cc7b98bc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65e8e1d29a1447bfa6bae10cc7b98bc6" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 33231 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:25.783937 412 master.cc:537] Master@127.0.103.61:44163 shutting down...
I20250111 02:51:25.790470 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:25.790750 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:25.790843 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2332d77de4da4b28b16a9f5ed89c700d: stopping tablet replica
I20250111 02:51:25.805524 412 master.cc:559] Master@127.0.103.61:44163 shutdown complete.
W20250111 02:51:26.737895 1866 heartbeater.cc:643] Failed to heartbeat to 127.0.103.61:44163 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.103.61:44163: connect: Connection refused (error 111)
I20250111 02:51:27.267213 2004 raft_consensus.cc:491] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2332d77de4da4b28b16a9f5ed89c700d)
I20250111 02:51:27.267410 2004 raft_consensus.cc:513] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:27.267966 2005 raft_consensus.cc:491] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2332d77de4da4b28b16a9f5ed89c700d)
I20250111 02:51:27.268069 2004 leader_election.cc:290] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12999f39682d496a8389b0b500d76d5a (127.0.103.62:37347), 2332d77de4da4b28b16a9f5ed89c700d (127.0.103.61:44163)
I20250111 02:51:27.268162 2005 raft_consensus.cc:513] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:27.268646 1504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "861c39487cb84dd8b88bebfc32b601c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12999f39682d496a8389b0b500d76d5a" is_pre_election: true
I20250111 02:51:27.268836 1504 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 861c39487cb84dd8b88bebfc32b601c8 in term 1.
I20250111 02:51:27.269114 2005 leader_election.cc:290] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2332d77de4da4b28b16a9f5ed89c700d (127.0.103.61:44163), 861c39487cb84dd8b88bebfc32b601c8 (127.0.103.60:41545)
I20250111 02:51:27.269383 1611 leader_election.cc:304] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12999f39682d496a8389b0b500d76d5a, 861c39487cb84dd8b88bebfc32b601c8; no voters:
I20250111 02:51:27.269686 2004 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:51:27.269892 2004 raft_consensus.cc:491] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2332d77de4da4b28b16a9f5ed89c700d)
I20250111 02:51:27.270022 2004 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:27.270371 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12999f39682d496a8389b0b500d76d5a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "861c39487cb84dd8b88bebfc32b601c8" is_pre_election: true
W20250111 02:51:27.270483 1609 leader_election.cc:336] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2332d77de4da4b28b16a9f5ed89c700d (127.0.103.61:44163): Network error: Client connection negotiation failed: client connection to 127.0.103.61:44163: connect: Connection refused (error 111)
I20250111 02:51:27.271979 2004 raft_consensus.cc:513] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:27.272308 1635 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12999f39682d496a8389b0b500d76d5a in current term 2: Already voted for candidate 861c39487cb84dd8b88bebfc32b601c8 in this term.
W20250111 02:51:27.273056 1479 leader_election.cc:336] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2332d77de4da4b28b16a9f5ed89c700d (127.0.103.61:44163): Network error: Client connection negotiation failed: client connection to 127.0.103.61:44163: connect: Connection refused (error 111)
I20250111 02:51:27.273241 1479 leader_election.cc:304] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12999f39682d496a8389b0b500d76d5a; no voters: 2332d77de4da4b28b16a9f5ed89c700d, 861c39487cb84dd8b88bebfc32b601c8
I20250111 02:51:27.273525 2005 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:51:27.273527 1504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "861c39487cb84dd8b88bebfc32b601c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12999f39682d496a8389b0b500d76d5a"
I20250111 02:51:27.273233 2004 leader_election.cc:290] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [CANDIDATE]: Term 2 election: Requested vote from peers 12999f39682d496a8389b0b500d76d5a (127.0.103.62:37347), 2332d77de4da4b28b16a9f5ed89c700d (127.0.103.61:44163)
W20250111 02:51:27.274434 1609 leader_election.cc:336] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2332d77de4da4b28b16a9f5ed89c700d (127.0.103.61:44163): Network error: Client connection negotiation failed: client connection to 127.0.103.61:44163: connect: Connection refused (error 111)
I20250111 02:51:27.275569 2005 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250111 02:51:27.277109 1504 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 861c39487cb84dd8b88bebfc32b601c8 in term 2.
I20250111 02:51:27.277546 1611 leader_election.cc:304] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12999f39682d496a8389b0b500d76d5a, 861c39487cb84dd8b88bebfc32b601c8; no voters: 2332d77de4da4b28b16a9f5ed89c700d
I20250111 02:51:27.277776 2004 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:51:27.278129 2004 raft_consensus.cc:695] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 2 LEADER]: Becoming Leader. State: Replica: 861c39487cb84dd8b88bebfc32b601c8, State: Running, Role: LEADER
I20250111 02:51:27.278393 2004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } }
I20250111 02:51:27.279515 2010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 861c39487cb84dd8b88bebfc32b601c8. Latest consensus state: current_term: 2 leader_uuid: "861c39487cb84dd8b88bebfc32b601c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:27.279744 2010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:27.280226 2012 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:51:27.281701 2012 catalog_manager.cc:670] Loaded metadata for table test-workload [id=d304fce92fa04aac9c5a9bbf91940251]
I20250111 02:51:27.282907 2012 tablet_loader.cc:96] loaded metadata for tablet 7731c2211e9e42b09d8737989e9ab575 (table test-workload [id=d304fce92fa04aac9c5a9bbf91940251])
I20250111 02:51:27.283196 2012 tablet_loader.cc:96] loaded metadata for tablet a7ef0cef0d5641268fe59dbd9449c8a4 (table test-workload [id=d304fce92fa04aac9c5a9bbf91940251])
I20250111 02:51:27.283438 2012 tablet_loader.cc:96] loaded metadata for tablet ccafb989d312485f91955d845e3e3920 (table test-workload [id=d304fce92fa04aac9c5a9bbf91940251])
I20250111 02:51:27.283607 2012 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:51:27.284304 2012 catalog_manager.cc:1260] Loaded cluster ID: 94244db43bb14435a62cf6e5d5b15395
I20250111 02:51:27.284380 2012 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:51:27.285392 2012 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:51:27.286242 2012 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: Loaded TSK: 0
I20250111 02:51:27.286816 2012 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:51:27.686051 1504 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 2 FOLLOWER]: Refusing update from remote peer 861c39487cb84dd8b88bebfc32b601c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250111 02:51:27.686640 2010 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250111 02:51:27.687988 2005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 861c39487cb84dd8b88bebfc32b601c8. Latest consensus state: current_term: 2 leader_uuid: "861c39487cb84dd8b88bebfc32b601c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:27.688220 2005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:27.690234 2005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "861c39487cb84dd8b88bebfc32b601c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:27.690496 2005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [sys.catalog]: This master's current role is: FOLLOWER
I20250111 02:51:27.690150 2004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "861c39487cb84dd8b88bebfc32b601c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12999f39682d496a8389b0b500d76d5a" member_type: VOTER last_known_addr { host: "127.0.103.62" port: 37347 } } peers { permanent_uuid: "2332d77de4da4b28b16a9f5ed89c700d" member_type: VOTER last_known_addr { host: "127.0.103.61" port: 44163 } } peers { permanent_uuid: "861c39487cb84dd8b88bebfc32b601c8" member_type: VOTER last_known_addr { host: "127.0.103.60" port: 41545 } } }
I20250111 02:51:27.690868 2004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [sys.catalog]: This master's current role is: LEADER
W20250111 02:51:27.691278 1609 consensus_peers.cc:487] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 -> Peer 2332d77de4da4b28b16a9f5ed89c700d (127.0.103.61:44163): Couldn't send request to peer 2332d77de4da4b28b16a9f5ed89c700d. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.61:44163: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:51:28.304442 1609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.61:44163: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250111 02:51:28.731112 1864 heartbeater.cc:502] Master 127.0.103.60:41545 was elected leader, sending a full tablet report...
I20250111 02:51:28.761425 1942 heartbeater.cc:502] Master 127.0.103.60:41545 was elected leader, sending a full tablet report...
I20250111 02:51:28.762037 1785 heartbeater.cc:502] Master 127.0.103.60:41545 was elected leader, sending a full tablet report...
I20250111 02:51:29.276121 1696 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:29.276245 1696 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:29.363974 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:51:29.372100 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:29.372356 412 tablet_replica.cc:331] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6: stopping tablet replica
I20250111 02:51:29.372579 412 raft_consensus.cc:2238] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:29.372749 412 raft_consensus.cc:2267] T 7731c2211e9e42b09d8737989e9ab575 P 65e8e1d29a1447bfa6bae10cc7b98bc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:29.387938 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:51:29.390602 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:51:29.398710 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:29.398965 412 tablet_replica.cc:331] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438: stopping tablet replica
I20250111 02:51:29.399175 412 raft_consensus.cc:2238] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:29.399353 412 raft_consensus.cc:2267] T ccafb989d312485f91955d845e3e3920 P bd3c17ccd9984434b1ca5179fb168438 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:29.413213 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:51:29.415640 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:51:29.424305 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:29.424540 412 tablet_replica.cc:331] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5: stopping tablet replica
I20250111 02:51:29.424721 412 raft_consensus.cc:2238] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:29.424896 412 raft_consensus.cc:2267] T a7ef0cef0d5641268fe59dbd9449c8a4 P ec851bd29c3140e7939076908d1fc8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:29.438427 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:51:29.440994 412 master.cc:537] Master@127.0.103.62:37347 shutting down...
I20250111 02:51:29.447978 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:29.448185 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:29.448287 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 12999f39682d496a8389b0b500d76d5a: stopping tablet replica
I20250111 02:51:29.461591 412 master.cc:559] Master@127.0.103.62:37347 shutdown complete.
I20250111 02:51:29.464895 412 master.cc:537] Master@127.0.103.60:41545 shutting down...
I20250111 02:51:29.471736 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:51:29.472091 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:29.472221 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 861c39487cb84dd8b88bebfc32b601c8: stopping tablet replica
I20250111 02:51:29.485800 412 master.cc:559] Master@127.0.103.60:41545 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6414 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20250111 02:51:29.503259 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:38221
I20250111 02:51:29.503705 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:29.506633 2027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:29.507678 2028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:29.508610 2030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:29.509375 412 server_base.cc:1034] running on GCE node
I20250111 02:51:29.509657 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:29.509708 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:29.509761 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563889509757 us; error 0 us; skew 500 ppm
I20250111 02:51:29.509932 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:29.511104 412 webserver.cc:458] Webserver started at http://127.0.103.62:36285/ using document root <none> and password file <none>
I20250111 02:51:29.511307 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:29.511387 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:29.511512 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:29.512077 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "6727c59a64e34d61a13b7bb67f469053"
format_stamp: "Formatted at 2025-01-11 02:51:29 on dist-test-slave-4xrj"
I20250111 02:51:29.514248 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:51:29.515499 2035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.515777 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:29.515878 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "6727c59a64e34d61a13b7bb67f469053"
format_stamp: "Formatted at 2025-01-11 02:51:29 on dist-test-slave-4xrj"
I20250111 02:51:29.515976 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:29.534430 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:29.534960 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:29.544842 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:38221
I20250111 02:51:29.544857 2086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:38221 every 8 connection(s)
I20250111 02:51:29.546043 2087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:29.549474 2087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053: Bootstrap starting.
I20250111 02:51:29.550711 2087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:29.552119 2087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053: No bootstrap required, opened a new log
I20250111 02:51:29.552629 2087 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6727c59a64e34d61a13b7bb67f469053" member_type: VOTER }
I20250111 02:51:29.552736 2087 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:29.552785 2087 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6727c59a64e34d61a13b7bb67f469053, State: Initialized, Role: FOLLOWER
I20250111 02:51:29.552951 2087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6727c59a64e34d61a13b7bb67f469053" member_type: VOTER }
I20250111 02:51:29.553049 2087 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:29.553105 2087 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:29.553177 2087 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:29.554610 2087 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6727c59a64e34d61a13b7bb67f469053" member_type: VOTER }
I20250111 02:51:29.554761 2087 leader_election.cc:304] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6727c59a64e34d61a13b7bb67f469053; no voters:
I20250111 02:51:29.555080 2087 leader_election.cc:290] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:29.555181 2090 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:29.555630 2090 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 1 LEADER]: Becoming Leader. State: Replica: 6727c59a64e34d61a13b7bb67f469053, State: Running, Role: LEADER
I20250111 02:51:29.555868 2087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:29.555828 2090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6727c59a64e34d61a13b7bb67f469053" member_type: VOTER }
I20250111 02:51:29.556563 2092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6727c59a64e34d61a13b7bb67f469053. Latest consensus state: current_term: 1 leader_uuid: "6727c59a64e34d61a13b7bb67f469053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6727c59a64e34d61a13b7bb67f469053" member_type: VOTER } }
I20250111 02:51:29.556689 2092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:29.556922 2091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6727c59a64e34d61a13b7bb67f469053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6727c59a64e34d61a13b7bb67f469053" member_type: VOTER } }
I20250111 02:51:29.557094 2091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:29.558079 2095 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:51:29.560006 2095 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:51:29.563208 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:51:29.563335 2095 catalog_manager.cc:1348] Generated new cluster ID: c100ed7e086146bb9172ec9f7976d40b
I20250111 02:51:29.563397 2095 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:51:29.583148 2095 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:51:29.583916 2095 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:51:29.592173 2095 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053: Generated new TSK 0
I20250111 02:51:29.592391 2095 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:51:29.595662 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:29.598938 2108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:29.599718 2109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:29.600719 2111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:29.601579 412 server_base.cc:1034] running on GCE node
I20250111 02:51:29.601814 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:29.601902 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:29.601941 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563889601937 us; error 0 us; skew 500 ppm
I20250111 02:51:29.602083 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:29.603173 412 webserver.cc:458] Webserver started at http://127.0.103.1:36965/ using document root <none> and password file <none>
I20250111 02:51:29.603379 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:29.603446 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:29.603572 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:29.604128 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "404f8228618e4807a25f947c6dbca9f8"
format_stamp: "Formatted at 2025-01-11 02:51:29 on dist-test-slave-4xrj"
I20250111 02:51:29.606417 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:51:29.607784 2116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.608071 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:29.608160 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "404f8228618e4807a25f947c6dbca9f8"
format_stamp: "Formatted at 2025-01-11 02:51:29 on dist-test-slave-4xrj"
I20250111 02:51:29.608256 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:29.620332 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:29.620795 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:29.621277 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:29.622076 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:29.622130 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.622200 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:29.622234 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.634366 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:36489
I20250111 02:51:29.634397 2178 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:36489 every 8 connection(s)
I20250111 02:51:29.635725 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:29.639057 2183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:29.639897 2184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:29.640849 2186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:29.641738 412 server_base.cc:1034] running on GCE node
I20250111 02:51:29.642019 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:29.642089 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:29.642124 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563889642119 us; error 0 us; skew 500 ppm
I20250111 02:51:29.642252 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:29.642277 2179 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38221
I20250111 02:51:29.642385 2179 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:29.642726 2179 heartbeater.cc:510] Master 127.0.103.62:38221 requested a full tablet report, sending...
I20250111 02:51:29.643513 412 webserver.cc:458] Webserver started at http://127.0.103.2:34703/ using document root <none> and password file <none>
I20250111 02:51:29.643669 2052 ts_manager.cc:194] Registered new tserver with Master: 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1:36489)
I20250111 02:51:29.643740 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:29.643806 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:29.643914 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:29.644629 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "923c145f7c3b4827857143b049c4c8cb"
format_stamp: "Formatted at 2025-01-11 02:51:29 on dist-test-slave-4xrj"
I20250111 02:51:29.645028 2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35602
I20250111 02:51:29.647257 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250111 02:51:29.648597 2191 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.648910 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:29.649003 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "923c145f7c3b4827857143b049c4c8cb"
format_stamp: "Formatted at 2025-01-11 02:51:29 on dist-test-slave-4xrj"
I20250111 02:51:29.649092 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:29.657217 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:29.657636 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:29.658130 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:29.658851 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:29.658905 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.658972 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:29.659008 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.670480 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:35755
I20250111 02:51:29.670506 2253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:35755 every 8 connection(s)
I20250111 02:51:29.671861 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:29.676395 2260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:29.677114 2257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:29.677121 2258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:29.677459 2254 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38221
I20250111 02:51:29.677564 2254 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:29.677886 2254 heartbeater.cc:510] Master 127.0.103.62:38221 requested a full tablet report, sending...
I20250111 02:51:29.677980 412 server_base.cc:1034] running on GCE node
I20250111 02:51:29.678256 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:29.678328 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:29.678372 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563889678368 us; error 0 us; skew 500 ppm
I20250111 02:51:29.678511 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:29.678743 2052 ts_manager.cc:194] Registered new tserver with Master: 923c145f7c3b4827857143b049c4c8cb (127.0.103.2:35755)
I20250111 02:51:29.679668 2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35604
I20250111 02:51:29.679706 412 webserver.cc:458] Webserver started at http://127.0.103.3:36425/ using document root <none> and password file <none>
I20250111 02:51:29.679898 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:29.679960 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:29.680079 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:29.680665 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "ed676450b28e4c669b94f3d9412adc29"
format_stamp: "Formatted at 2025-01-11 02:51:29 on dist-test-slave-4xrj"
I20250111 02:51:29.682804 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:51:29.684172 2265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.684439 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:51:29.684525 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "ed676450b28e4c669b94f3d9412adc29"
format_stamp: "Formatted at 2025-01-11 02:51:29 on dist-test-slave-4xrj"
I20250111 02:51:29.684610 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:29.696316 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:29.696767 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:29.697207 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:29.697896 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:29.697944 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.698004 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:29.698036 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:29.709594 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:33241
I20250111 02:51:29.709615 2327 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:33241 every 8 connection(s)
I20250111 02:51:29.715433 2328 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38221
I20250111 02:51:29.715535 2328 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:29.715797 2328 heartbeater.cc:510] Master 127.0.103.62:38221 requested a full tablet report, sending...
I20250111 02:51:29.716454 2052 ts_manager.cc:194] Registered new tserver with Master: ed676450b28e4c669b94f3d9412adc29 (127.0.103.3:33241)
I20250111 02:51:29.717051 2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35606
I20250111 02:51:29.717197 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006798967s
I20250111 02:51:30.646464 2179 heartbeater.cc:502] Master 127.0.103.62:38221 was elected leader, sending a full tablet report...
I20250111 02:51:30.680826 2254 heartbeater.cc:502] Master 127.0.103.62:38221 was elected leader, sending a full tablet report...
I20250111 02:51:30.718267 2328 heartbeater.cc:502] Master 127.0.103.62:38221 was elected leader, sending a full tablet report...
I20250111 02:51:30.744747 412 test_util.cc:274] Using random seed: 1238789541
I20250111 02:51:30.758821 2052 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35608:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:51:30.759402 2052 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:51:30.787257 2219 tablet_service.cc:1467] Processing CreateTablet for tablet 49787e8a20f842e886ff576a636bb1b3 (DEFAULT_TABLE table=test-workload [id=a46397c5501b4a15b5ca619da10d056c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250111 02:51:30.787314 2144 tablet_service.cc:1467] Processing CreateTablet for tablet 49787e8a20f842e886ff576a636bb1b3 (DEFAULT_TABLE table=test-workload [id=a46397c5501b4a15b5ca619da10d056c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250111 02:51:30.787703 2219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49787e8a20f842e886ff576a636bb1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:30.787750 2144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49787e8a20f842e886ff576a636bb1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:30.788650 2143 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfdc94fe6354925bdf24b1e2399f178 (DEFAULT_TABLE table=test-workload [id=a46397c5501b4a15b5ca619da10d056c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250111 02:51:30.789054 2143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfdc94fe6354925bdf24b1e2399f178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:30.792552 2348 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb: Bootstrap starting.
I20250111 02:51:30.792505 2218 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfdc94fe6354925bdf24b1e2399f178 (DEFAULT_TABLE table=test-workload [id=a46397c5501b4a15b5ca619da10d056c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250111 02:51:30.792876 2218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfdc94fe6354925bdf24b1e2399f178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:30.794355 2348 tablet_bootstrap.cc:654] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:30.795578 2349 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: Bootstrap starting.
I20250111 02:51:30.796904 2349 tablet_bootstrap.cc:654] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:30.799474 2293 tablet_service.cc:1467] Processing CreateTablet for tablet 49787e8a20f842e886ff576a636bb1b3 (DEFAULT_TABLE table=test-workload [id=a46397c5501b4a15b5ca619da10d056c]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250111 02:51:30.799602 2349 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: No bootstrap required, opened a new log
I20250111 02:51:30.799712 2349 ts_tablet_manager.cc:1397] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250111 02:51:30.799851 2293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49787e8a20f842e886ff576a636bb1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:30.800319 2349 raft_consensus.cc:357] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.800470 2349 raft_consensus.cc:383] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:30.800575 2349 raft_consensus.cc:738] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404f8228618e4807a25f947c6dbca9f8, State: Initialized, Role: FOLLOWER
I20250111 02:51:30.800814 2349 consensus_queue.cc:260] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.801491 2349 ts_tablet_manager.cc:1428] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:51:30.801769 2349 tablet_bootstrap.cc:492] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8: Bootstrap starting.
I20250111 02:51:30.802167 2292 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfdc94fe6354925bdf24b1e2399f178 (DEFAULT_TABLE table=test-workload [id=a46397c5501b4a15b5ca619da10d056c]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250111 02:51:30.802587 2292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfdc94fe6354925bdf24b1e2399f178. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:30.803508 2349 tablet_bootstrap.cc:654] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:30.806500 2353 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29: Bootstrap starting.
I20250111 02:51:30.806556 2348 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb: No bootstrap required, opened a new log
I20250111 02:51:30.807186 2348 ts_tablet_manager.cc:1397] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20250111 02:51:30.807967 2348 raft_consensus.cc:357] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.808257 2348 raft_consensus.cc:383] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:30.808429 2348 raft_consensus.cc:738] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923c145f7c3b4827857143b049c4c8cb, State: Initialized, Role: FOLLOWER
I20250111 02:51:30.808704 2353 tablet_bootstrap.cc:654] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:30.808724 2348 consensus_queue.cc:260] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.809868 2349 tablet_bootstrap.cc:492] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8: No bootstrap required, opened a new log
I20250111 02:51:30.810099 2349 ts_tablet_manager.cc:1397] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20250111 02:51:30.810322 2348 ts_tablet_manager.cc:1428] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250111 02:51:30.810636 2348 tablet_bootstrap.cc:492] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb: Bootstrap starting.
I20250111 02:51:30.811977 2348 tablet_bootstrap.cc:654] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:30.811944 2349 raft_consensus.cc:357] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.812350 2349 raft_consensus.cc:383] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:30.812512 2349 raft_consensus.cc:738] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404f8228618e4807a25f947c6dbca9f8, State: Initialized, Role: FOLLOWER
I20250111 02:51:30.812848 2349 consensus_queue.cc:260] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.813448 2349 ts_tablet_manager.cc:1428] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250111 02:51:30.814193 2348 tablet_bootstrap.cc:492] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb: No bootstrap required, opened a new log
I20250111 02:51:30.814384 2348 ts_tablet_manager.cc:1397] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250111 02:51:30.815136 2348 raft_consensus.cc:357] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.815361 2348 raft_consensus.cc:383] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:30.815459 2348 raft_consensus.cc:738] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 923c145f7c3b4827857143b049c4c8cb, State: Initialized, Role: FOLLOWER
I20250111 02:51:30.815742 2353 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29: No bootstrap required, opened a new log
I20250111 02:51:30.815707 2348 consensus_queue.cc:260] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.815913 2353 ts_tablet_manager.cc:1397] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20250111 02:51:30.816279 2348 ts_tablet_manager.cc:1428] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:51:30.816628 2353 raft_consensus.cc:357] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.816838 2353 raft_consensus.cc:383] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:30.816941 2353 raft_consensus.cc:738] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed676450b28e4c669b94f3d9412adc29, State: Initialized, Role: FOLLOWER
I20250111 02:51:30.817183 2353 consensus_queue.cc:260] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.817910 2353 ts_tablet_manager.cc:1428] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250111 02:51:30.818218 2353 tablet_bootstrap.cc:492] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29: Bootstrap starting.
I20250111 02:51:30.819605 2353 tablet_bootstrap.cc:654] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:30.821758 2353 tablet_bootstrap.cc:492] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29: No bootstrap required, opened a new log
I20250111 02:51:30.821900 2353 ts_tablet_manager.cc:1397] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250111 02:51:30.822376 2353 raft_consensus.cc:357] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.822510 2353 raft_consensus.cc:383] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:30.822558 2353 raft_consensus.cc:738] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed676450b28e4c669b94f3d9412adc29, State: Initialized, Role: FOLLOWER
I20250111 02:51:30.822747 2353 consensus_queue.cc:260] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.823091 2353 ts_tablet_manager.cc:1428] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:51:30.839888 2355 raft_consensus.cc:491] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:30.840035 2355 raft_consensus.cc:513] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.840806 2355 leader_election.cc:290] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1:36489), ed676450b28e4c669b94f3d9412adc29 (127.0.103.3:33241)
I20250111 02:51:30.846431 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfdc94fe6354925bdf24b1e2399f178" candidate_uuid: "923c145f7c3b4827857143b049c4c8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed676450b28e4c669b94f3d9412adc29" is_pre_election: true
I20250111 02:51:30.846472 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfdc94fe6354925bdf24b1e2399f178" candidate_uuid: "923c145f7c3b4827857143b049c4c8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404f8228618e4807a25f947c6dbca9f8" is_pre_election: true
I20250111 02:51:30.846632 2303 raft_consensus.cc:2463] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923c145f7c3b4827857143b049c4c8cb in term 0.
I20250111 02:51:30.846696 2154 raft_consensus.cc:2463] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 923c145f7c3b4827857143b049c4c8cb in term 0.
I20250111 02:51:30.847035 2194 leader_election.cc:304] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923c145f7c3b4827857143b049c4c8cb, ed676450b28e4c669b94f3d9412adc29; no voters:
I20250111 02:51:30.847260 2355 raft_consensus.cc:2798] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:30.847380 2355 raft_consensus.cc:491] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:30.847455 2355 raft_consensus.cc:3054] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:30.849040 2355 raft_consensus.cc:513] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.849495 2355 leader_election.cc:290] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [CANDIDATE]: Term 1 election: Requested vote from peers 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1:36489), ed676450b28e4c669b94f3d9412adc29 (127.0.103.3:33241)
I20250111 02:51:30.849831 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfdc94fe6354925bdf24b1e2399f178" candidate_uuid: "923c145f7c3b4827857143b049c4c8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404f8228618e4807a25f947c6dbca9f8"
I20250111 02:51:30.849974 2303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cfdc94fe6354925bdf24b1e2399f178" candidate_uuid: "923c145f7c3b4827857143b049c4c8cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed676450b28e4c669b94f3d9412adc29"
I20250111 02:51:30.850510 2303 raft_consensus.cc:3054] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:30.850535 2154 raft_consensus.cc:3054] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:30.852084 2303 raft_consensus.cc:2463] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923c145f7c3b4827857143b049c4c8cb in term 1.
I20250111 02:51:30.852171 2154 raft_consensus.cc:2463] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 923c145f7c3b4827857143b049c4c8cb in term 1.
I20250111 02:51:30.852406 2194 leader_election.cc:304] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 923c145f7c3b4827857143b049c4c8cb, ed676450b28e4c669b94f3d9412adc29; no voters:
I20250111 02:51:30.852651 2355 raft_consensus.cc:2798] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:30.853029 2355 raft_consensus.cc:695] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 1 LEADER]: Becoming Leader. State: Replica: 923c145f7c3b4827857143b049c4c8cb, State: Running, Role: LEADER
I20250111 02:51:30.853289 2355 consensus_queue.cc:237] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:30.855706 2051 catalog_manager.cc:5526] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 923c145f7c3b4827857143b049c4c8cb (127.0.103.2). New cstate: current_term: 1 leader_uuid: "923c145f7c3b4827857143b049c4c8cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:31.035068 2358 raft_consensus.cc:491] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:51:31.035234 2358 raft_consensus.cc:513] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:31.036000 2358 leader_election.cc:290] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 923c145f7c3b4827857143b049c4c8cb (127.0.103.2:35755), 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1:36489)
I20250111 02:51:31.041361 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49787e8a20f842e886ff576a636bb1b3" candidate_uuid: "ed676450b28e4c669b94f3d9412adc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404f8228618e4807a25f947c6dbca9f8" is_pre_election: true
I20250111 02:51:31.041445 2229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49787e8a20f842e886ff576a636bb1b3" candidate_uuid: "ed676450b28e4c669b94f3d9412adc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923c145f7c3b4827857143b049c4c8cb" is_pre_election: true
I20250111 02:51:31.041680 2154 raft_consensus.cc:2463] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed676450b28e4c669b94f3d9412adc29 in term 0.
I20250111 02:51:31.041697 2229 raft_consensus.cc:2463] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed676450b28e4c669b94f3d9412adc29 in term 0.
I20250111 02:51:31.042060 2266 leader_election.cc:304] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 404f8228618e4807a25f947c6dbca9f8, ed676450b28e4c669b94f3d9412adc29; no voters:
I20250111 02:51:31.042307 2358 raft_consensus.cc:2798] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:51:31.042402 2358 raft_consensus.cc:491] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:51:31.042496 2358 raft_consensus.cc:3054] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:31.043957 2358 raft_consensus.cc:513] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:31.044438 2358 leader_election.cc:290] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [CANDIDATE]: Term 1 election: Requested vote from peers 923c145f7c3b4827857143b049c4c8cb (127.0.103.2:35755), 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1:36489)
I20250111 02:51:31.044742 2229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49787e8a20f842e886ff576a636bb1b3" candidate_uuid: "ed676450b28e4c669b94f3d9412adc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "923c145f7c3b4827857143b049c4c8cb"
I20250111 02:51:31.044814 2154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49787e8a20f842e886ff576a636bb1b3" candidate_uuid: "ed676450b28e4c669b94f3d9412adc29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404f8228618e4807a25f947c6dbca9f8"
I20250111 02:51:31.044888 2229 raft_consensus.cc:3054] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:31.044992 2154 raft_consensus.cc:3054] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:31.047186 2154 raft_consensus.cc:2463] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed676450b28e4c669b94f3d9412adc29 in term 1.
I20250111 02:51:31.047343 2229 raft_consensus.cc:2463] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed676450b28e4c669b94f3d9412adc29 in term 1.
I20250111 02:51:31.047573 2266 leader_election.cc:304] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 404f8228618e4807a25f947c6dbca9f8, ed676450b28e4c669b94f3d9412adc29; no voters:
I20250111 02:51:31.047823 2358 raft_consensus.cc:2798] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:31.048148 2358 raft_consensus.cc:695] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 1 LEADER]: Becoming Leader. State: Replica: ed676450b28e4c669b94f3d9412adc29, State: Running, Role: LEADER
I20250111 02:51:31.048465 2358 consensus_queue.cc:237] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } }
I20250111 02:51:31.050704 2051 catalog_manager.cc:5526] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed676450b28e4c669b94f3d9412adc29 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "ed676450b28e4c669b94f3d9412adc29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } health_report { overall_health: HEALTHY } } }
I20250111 02:51:31.353194 2355 consensus_queue.cc:1035] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:31.357415 2370 consensus_queue.cc:1035] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:31.563164 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:31.563717 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:31.563858 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:31.644559 2358 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:31.650236 2376 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:51:33.564350 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:33.564895 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:33.564998 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:35.565418 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:35.566031 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:35.566172 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:37.566579 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:37.567211 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:37.567328 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:39.567786 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:39.568387 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:39.568545 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:41.569052 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:41.569669 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:41.569782 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:42.125308 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:42.129110 2386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:42.129920 2387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:42.131151 2389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:42.132140 412 server_base.cc:1034] running on GCE node
I20250111 02:51:42.132608 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:42.132686 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:42.132732 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563902132718 us; error 0 us; skew 500 ppm
I20250111 02:51:42.132860 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:42.134191 412 webserver.cc:458] Webserver started at http://127.0.103.4:44463/ using document root <none> and password file <none>
I20250111 02:51:42.134413 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:42.134487 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:42.134601 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:42.135268 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-3-root/instance:
uuid: "4e6a5a566c5c475782572f2f9377e089"
format_stamp: "Formatted at 2025-01-11 02:51:42 on dist-test-slave-4xrj"
I20250111 02:51:42.137651 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250111 02:51:42.139223 2394 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:42.139597 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:51:42.139706 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-3-root
uuid: "4e6a5a566c5c475782572f2f9377e089"
format_stamp: "Formatted at 2025-01-11 02:51:42 on dist-test-slave-4xrj"
I20250111 02:51:42.139801 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:42.156898 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:42.157445 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:42.157979 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:51:42.158752 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:51:42.158825 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:42.158895 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:51:42.158950 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:42.174129 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.4:44665
I20250111 02:51:42.174163 2456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.4:44665 every 8 connection(s)
I20250111 02:51:42.181069 2457 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38221
I20250111 02:51:42.181201 2457 heartbeater.cc:463] Registering TS with master...
I20250111 02:51:42.181579 2457 heartbeater.cc:510] Master 127.0.103.62:38221 requested a full tablet report, sending...
I20250111 02:51:42.182375 2052 ts_manager.cc:194] Registered new tserver with Master: 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4:44665)
I20250111 02:51:42.183044 2052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34422
I20250111 02:51:42.583253 2303 consensus_queue.cc:237] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: true } }
I20250111 02:51:42.585834 2229 raft_consensus.cc:1270] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Refusing update from remote peer ed676450b28e4c669b94f3d9412adc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:51:42.586285 2154 raft_consensus.cc:1270] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Refusing update from remote peer ed676450b28e4c669b94f3d9412adc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:51:42.587013 2385 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250111 02:51:42.587365 2464 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:51:42.590060 2385 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: true } } }
W20250111 02:51:42.592975 2269 consensus_peers.cc:487] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 -> Peer 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4:44665): Couldn't send request to peer 4e6a5a566c5c475782572f2f9377e089. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49787e8a20f842e886ff576a636bb1b3. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:51:42.594405 2154 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: true } } }
I20250111 02:51:42.595182 2229 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: true } } }
I20250111 02:51:42.596684 2052 catalog_manager.cc:5526] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 reported cstate change: config changed from index -1 to 2, NON_VOTER 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "ed676450b28e4c669b94f3d9412adc29" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:51:43.136234 2475 ts_tablet_manager.cc:927] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: Initiating tablet copy from peer ed676450b28e4c669b94f3d9412adc29 (127.0.103.3:33241)
I20250111 02:51:43.137172 2475 tablet_copy_client.cc:323] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:33241
I20250111 02:51:43.144559 2313 tablet_copy_service.cc:140] P ed676450b28e4c669b94f3d9412adc29: Received BeginTabletCopySession request for tablet 49787e8a20f842e886ff576a636bb1b3 from peer 4e6a5a566c5c475782572f2f9377e089 ({username='slave'} at 127.0.0.1:49008)
I20250111 02:51:43.144745 2313 tablet_copy_service.cc:161] P ed676450b28e4c669b94f3d9412adc29: Beginning new tablet copy session on tablet 49787e8a20f842e886ff576a636bb1b3 from peer 4e6a5a566c5c475782572f2f9377e089 at {username='slave'} at 127.0.0.1:49008: session id = 4e6a5a566c5c475782572f2f9377e089-49787e8a20f842e886ff576a636bb1b3
I20250111 02:51:43.147656 2313 tablet_copy_source_session.cc:215] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:51:43.149349 2475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49787e8a20f842e886ff576a636bb1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:43.154990 2475 tablet_copy_client.cc:806] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: tablet copy: Starting download of 0 data blocks...
I20250111 02:51:43.155282 2475 tablet_copy_client.cc:670] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: tablet copy: Starting download of 1 WAL segments...
I20250111 02:51:43.157156 2475 tablet_copy_client.cc:538] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:51:43.160758 2475 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: Bootstrap starting.
I20250111 02:51:43.168534 2475 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:51:43.169001 2475 tablet_bootstrap.cc:492] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: Bootstrap complete.
I20250111 02:51:43.169268 2475 ts_tablet_manager.cc:1397] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20250111 02:51:43.169991 2475 raft_consensus.cc:357] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: true } }
I20250111 02:51:43.170228 2475 raft_consensus.cc:738] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e6a5a566c5c475782572f2f9377e089, State: Initialized, Role: LEARNER
I20250111 02:51:43.170451 2475 consensus_queue.cc:260] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: true } }
I20250111 02:51:43.171140 2457 heartbeater.cc:502] Master 127.0.103.62:38221 was elected leader, sending a full tablet report...
I20250111 02:51:43.171762 2475 ts_tablet_manager.cc:1428] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: Time spent starting tablet: real 0.002s user 0.002s sys 0.003s
I20250111 02:51:43.172379 2313 tablet_copy_service.cc:342] P ed676450b28e4c669b94f3d9412adc29: Request end of tablet copy session 4e6a5a566c5c475782572f2f9377e089-49787e8a20f842e886ff576a636bb1b3 received from {username='slave'} at 127.0.0.1:49008
I20250111 02:51:43.172557 2313 tablet_copy_service.cc:434] P ed676450b28e4c669b94f3d9412adc29: ending tablet copy session 4e6a5a566c5c475782572f2f9377e089-49787e8a20f842e886ff576a636bb1b3 on tablet 49787e8a20f842e886ff576a636bb1b3 with peer 4e6a5a566c5c475782572f2f9377e089
I20250111 02:51:43.570043 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:43.570667 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:43.570866 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:43.675128 2432 raft_consensus.cc:1212] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250111 02:51:44.062026 2481 raft_consensus.cc:1059] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29: attempting to promote NON_VOTER 4e6a5a566c5c475782572f2f9377e089 to VOTER
I20250111 02:51:44.062892 2481 consensus_queue.cc:237] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } }
I20250111 02:51:44.064888 2154 raft_consensus.cc:1270] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Refusing update from remote peer ed676450b28e4c669b94f3d9412adc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:51:44.065065 2432 raft_consensus.cc:1270] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 LEARNER]: Refusing update from remote peer ed676450b28e4c669b94f3d9412adc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:51:44.065380 2229 raft_consensus.cc:1270] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Refusing update from remote peer ed676450b28e4c669b94f3d9412adc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:51:44.065654 2385 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:51:44.065896 2482 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:51:44.066076 2481 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:51:44.070025 2481 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.070637 2432 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.070693 2154 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.074167 2229 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.075253 2051 catalog_manager.cc:5526] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 reported cstate change: config changed from index 2 to 3, 4e6a5a566c5c475782572f2f9377e089 (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed676450b28e4c669b94f3d9412adc29" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "404f8228618e4807a25f947c6dbca9f8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36489 } attrs { replace: true } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.082224 2302 consensus_queue.cc:237] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } }
I20250111 02:51:44.083849 2229 raft_consensus.cc:1270] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Refusing update from remote peer ed676450b28e4c669b94f3d9412adc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:51:44.084028 2432 raft_consensus.cc:1270] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 FOLLOWER]: Refusing update from remote peer ed676450b28e4c669b94f3d9412adc29: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:51:44.084447 2482 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:51:44.085333 2482 consensus_queue.cc:1035] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:51:44.087018 2481 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.087558 2432 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.090211 2229 raft_consensus.cc:2949] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.091038 2050 catalog_manager.cc:5526] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 reported cstate change: config changed from index 3 to 4, VOTER 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1) evicted. New cstate: current_term: 1 leader_uuid: "ed676450b28e4c669b94f3d9412adc29" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "923c145f7c3b4827857143b049c4c8cb" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 35755 } } peers { permanent_uuid: "ed676450b28e4c669b94f3d9412adc29" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33241 } } peers { permanent_uuid: "4e6a5a566c5c475782572f2f9377e089" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 44665 } attrs { promote: false } } }
I20250111 02:51:44.092877 2038 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 49787e8a20f842e886ff576a636bb1b3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:51:44.094485 2144 tablet_service.cc:1514] Processing DeleteTablet for tablet 49787e8a20f842e886ff576a636bb1b3 with delete_type TABLET_DATA_TOMBSTONED (TS 404f8228618e4807a25f947c6dbca9f8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49006
I20250111 02:51:44.095430 2493 tablet_replica.cc:331] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: stopping tablet replica
I20250111 02:51:44.095602 2493 raft_consensus.cc:2238] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:44.095872 2493 raft_consensus.cc:2267] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:44.096702 2493 ts_tablet_manager.cc:1905] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:51:44.100211 2493 ts_tablet_manager.cc:1918] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250111 02:51:44.100304 2493 log.cc:1198] T 49787e8a20f842e886ff576a636bb1b3 P 404f8228618e4807a25f947c6dbca9f8: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1736563872491422-412-0/minicluster-data/ts-0-root/wals/49787e8a20f842e886ff576a636bb1b3
I20250111 02:51:44.100939 2036 catalog_manager.cc:4872] TS 404f8228618e4807a25f947c6dbca9f8 (127.0.103.1:36489): tablet 49787e8a20f842e886ff576a636bb1b3 (table test-workload [id=a46397c5501b4a15b5ca619da10d056c]) successfully deleted
I20250111 02:51:45.571372 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:45.571887 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:45.571972 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:47.572358 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:47.572912 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:47.573002 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:49.573459 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:49.573966 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:49.574057 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:51.574414 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:51.574955 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:51.575053 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:53.575553 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:53.576095 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:53.576215 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:55.576568 2106 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:51:55.577122 2106 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:51:55.577231 2106 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:51:56.259733 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:51:56.269376 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:56.269634 412 tablet_replica.cc:331] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8: stopping tablet replica
I20250111 02:51:56.269807 412 raft_consensus.cc:2238] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:56.270015 412 raft_consensus.cc:2267] T 5cfdc94fe6354925bdf24b1e2399f178 P 404f8228618e4807a25f947c6dbca9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:56.285954 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:51:56.288769 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:51:56.298069 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:56.298329 412 tablet_replica.cc:331] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb: stopping tablet replica
I20250111 02:51:56.298522 412 raft_consensus.cc:2238] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:56.298802 412 raft_consensus.cc:2267] T 5cfdc94fe6354925bdf24b1e2399f178 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:56.299361 412 tablet_replica.cc:331] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb: stopping tablet replica
I20250111 02:51:56.299472 412 raft_consensus.cc:2238] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:56.299649 412 raft_consensus.cc:2267] T 49787e8a20f842e886ff576a636bb1b3 P 923c145f7c3b4827857143b049c4c8cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:56.312923 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:51:56.315666 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:51:56.329250 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:56.329483 412 tablet_replica.cc:331] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29: stopping tablet replica
I20250111 02:51:56.329663 412 raft_consensus.cc:2238] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:56.329818 412 raft_consensus.cc:2267] T 5cfdc94fe6354925bdf24b1e2399f178 P ed676450b28e4c669b94f3d9412adc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:56.330328 412 tablet_replica.cc:331] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29: stopping tablet replica
I20250111 02:51:56.330438 412 raft_consensus.cc:2238] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:56.330649 412 raft_consensus.cc:2267] T 49787e8a20f842e886ff576a636bb1b3 P ed676450b28e4c669b94f3d9412adc29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:56.343868 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:51:56.346715 412 tablet_server.cc:178] TabletServer@127.0.103.4:0 shutting down...
I20250111 02:51:56.355299 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:51:56.355496 412 tablet_replica.cc:331] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089: stopping tablet replica
I20250111 02:51:56.355651 412 raft_consensus.cc:2238] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:51:56.355865 412 raft_consensus.cc:2267] T 49787e8a20f842e886ff576a636bb1b3 P 4e6a5a566c5c475782572f2f9377e089 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:56.368839 412 tablet_server.cc:195] TabletServer@127.0.103.4:0 shutdown complete.
I20250111 02:51:56.371462 412 master.cc:537] Master@127.0.103.62:38221 shutting down...
I20250111 02:51:56.378209 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:51:56.378422 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:51:56.378528 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6727c59a64e34d61a13b7bb67f469053: stopping tablet replica
I20250111 02:51:56.391358 412 master.cc:559] Master@127.0.103.62:38221 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26899 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20250111 02:51:56.402427 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:36137
I20250111 02:51:56.402832 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:51:56.405997 2499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:56.406934 2500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:51:56.408061 2502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:51:56.409024 412 server_base.cc:1034] running on GCE node
I20250111 02:51:56.409231 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:51:56.409279 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:51:56.409313 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563916409308 us; error 0 us; skew 500 ppm
I20250111 02:51:56.409456 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:51:56.415810 412 webserver.cc:458] Webserver started at http://127.0.103.62:42623/ using document root <none> and password file <none>
I20250111 02:51:56.416033 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:51:56.416098 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:51:56.416222 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:51:56.416822 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "949da0018283466ca6dd11e3049063f3"
format_stamp: "Formatted at 2025-01-11 02:51:56 on dist-test-slave-4xrj"
I20250111 02:51:56.419171 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:51:56.420552 2507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:51:56.420899 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:51:56.420979 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "949da0018283466ca6dd11e3049063f3"
format_stamp: "Formatted at 2025-01-11 02:51:56 on dist-test-slave-4xrj"
I20250111 02:51:56.421072 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:51:56.434996 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:51:56.435428 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:51:56.444895 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:36137
I20250111 02:51:56.444916 2558 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:36137 every 8 connection(s)
I20250111 02:51:56.446166 2559 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:51:56.449615 2559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3: Bootstrap starting.
I20250111 02:51:56.450666 2559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3: Neither blocks nor log segments found. Creating new log.
I20250111 02:51:56.452057 2559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3: No bootstrap required, opened a new log
I20250111 02:51:56.452616 2559 raft_consensus.cc:357] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949da0018283466ca6dd11e3049063f3" member_type: VOTER }
I20250111 02:51:56.452730 2559 raft_consensus.cc:383] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:51:56.452786 2559 raft_consensus.cc:738] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 949da0018283466ca6dd11e3049063f3, State: Initialized, Role: FOLLOWER
I20250111 02:51:56.452965 2559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949da0018283466ca6dd11e3049063f3" member_type: VOTER }
I20250111 02:51:56.453068 2559 raft_consensus.cc:397] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:51:56.453125 2559 raft_consensus.cc:491] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:51:56.453198 2559 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:51:56.454699 2559 raft_consensus.cc:513] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949da0018283466ca6dd11e3049063f3" member_type: VOTER }
I20250111 02:51:56.454861 2559 leader_election.cc:304] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 949da0018283466ca6dd11e3049063f3; no voters:
I20250111 02:51:56.455199 2559 leader_election.cc:290] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:51:56.455277 2562 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:51:56.455747 2562 raft_consensus.cc:695] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 1 LEADER]: Becoming Leader. State: Replica: 949da0018283466ca6dd11e3049063f3, State: Running, Role: LEADER
I20250111 02:51:56.455945 2562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949da0018283466ca6dd11e3049063f3" member_type: VOTER }
I20250111 02:51:56.456005 2559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:51:56.456552 2563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "949da0018283466ca6dd11e3049063f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949da0018283466ca6dd11e3049063f3" member_type: VOTER } }
I20250111 02:51:56.456619 2564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 949da0018283466ca6dd11e3049063f3. Latest consensus state: current_term: 1 leader_uuid: "949da0018283466ca6dd11e3049063f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "949da0018283466ca6dd11e3049063f3" member_type: VOTER } }
I20250111 02:51:56.456768 2564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:56.457729 2563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [sys.catalog]: This master's current role is: LEADER
I20250111 02:51:56.457870 2566 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:51:56.459298 2566 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:51:56.461732 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:51:56.462010 2566 catalog_manager.cc:1348] Generated new cluster ID: 9d09db2f06e14ef3a42cefefd54db85d
I20250111 02:51:56.462083 2566 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:51:56.479102 2566 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:51:56.479799 2566 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:51:56.497871 2566 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3: Generated new TSK 0
I20250111 02:51:56.498119 2566 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:51:56.525985 412 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 7.806e-05s
I20250111 02:52:08.611642 412 master.cc:537] Master@127.0.103.62:36137 shutting down...
I20250111 02:52:08.624786 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:52:08.625007 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:08.625085 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 949da0018283466ca6dd11e3049063f3: stopping tablet replica
I20250111 02:52:08.643275 412 master.cc:559] Master@127.0.103.62:36137 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12255 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20250111 02:52:08.657420 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:44347
I20250111 02:52:08.657825 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:08.661126 2581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:08.661963 2582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:08.662992 2584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:08.663904 412 server_base.cc:1034] running on GCE node
I20250111 02:52:08.664109 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:08.664160 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:08.664192 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563928664189 us; error 0 us; skew 500 ppm
I20250111 02:52:08.664340 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:08.665480 412 webserver.cc:458] Webserver started at http://127.0.103.62:37129/ using document root <none> and password file <none>
I20250111 02:52:08.665664 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:08.665724 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:08.665876 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:08.666468 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "28b10236cd074f7a8c3252a5890bff98"
format_stamp: "Formatted at 2025-01-11 02:52:08 on dist-test-slave-4xrj"
I20250111 02:52:08.668793 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:08.670189 2589 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.670477 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:08.670578 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "28b10236cd074f7a8c3252a5890bff98"
format_stamp: "Formatted at 2025-01-11 02:52:08 on dist-test-slave-4xrj"
I20250111 02:52:08.670670 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:08.680485 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:08.680891 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:08.689875 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:44347
I20250111 02:52:08.689893 2640 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:44347 every 8 connection(s)
I20250111 02:52:08.690938 2641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:08.694358 2641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98: Bootstrap starting.
I20250111 02:52:08.695488 2641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:08.696802 2641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98: No bootstrap required, opened a new log
I20250111 02:52:08.697268 2641 raft_consensus.cc:357] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b10236cd074f7a8c3252a5890bff98" member_type: VOTER }
I20250111 02:52:08.697376 2641 raft_consensus.cc:383] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:08.697412 2641 raft_consensus.cc:738] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b10236cd074f7a8c3252a5890bff98, State: Initialized, Role: FOLLOWER
I20250111 02:52:08.697561 2641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b10236cd074f7a8c3252a5890bff98" member_type: VOTER }
I20250111 02:52:08.697655 2641 raft_consensus.cc:397] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:52:08.697691 2641 raft_consensus.cc:491] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:52:08.697736 2641 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:08.699065 2641 raft_consensus.cc:513] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b10236cd074f7a8c3252a5890bff98" member_type: VOTER }
I20250111 02:52:08.699203 2641 leader_election.cc:304] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b10236cd074f7a8c3252a5890bff98; no voters:
I20250111 02:52:08.699476 2641 leader_election.cc:290] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:52:08.699555 2644 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:08.699903 2644 raft_consensus.cc:695] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 1 LEADER]: Becoming Leader. State: Replica: 28b10236cd074f7a8c3252a5890bff98, State: Running, Role: LEADER
I20250111 02:52:08.700098 2644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b10236cd074f7a8c3252a5890bff98" member_type: VOTER }
I20250111 02:52:08.700150 2641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:52:08.700685 2645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28b10236cd074f7a8c3252a5890bff98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b10236cd074f7a8c3252a5890bff98" member_type: VOTER } }
I20250111 02:52:08.700736 2646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b10236cd074f7a8c3252a5890bff98. Latest consensus state: current_term: 1 leader_uuid: "28b10236cd074f7a8c3252a5890bff98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b10236cd074f7a8c3252a5890bff98" member_type: VOTER } }
I20250111 02:52:08.700911 2646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [sys.catalog]: This master's current role is: LEADER
I20250111 02:52:08.700901 2645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [sys.catalog]: This master's current role is: LEADER
I20250111 02:52:08.701784 2649 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:52:08.702885 2649 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:52:08.705598 2649 catalog_manager.cc:1348] Generated new cluster ID: f8f670697d9c44bdbdd8b7a4f232e4dc
I20250111 02:52:08.705668 2649 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:52:08.706804 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:52:08.723259 2649 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:52:08.723982 2649 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:52:08.730659 2649 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98: Generated new TSK 0
I20250111 02:52:08.730883 2649 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:52:08.739187 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:08.742360 2662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:08.743487 2663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:08.744431 2665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:08.745285 412 server_base.cc:1034] running on GCE node
I20250111 02:52:08.745509 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:08.745558 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:08.745594 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563928745588 us; error 0 us; skew 500 ppm
I20250111 02:52:08.745723 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:08.746920 412 webserver.cc:458] Webserver started at http://127.0.103.1:39475/ using document root <none> and password file <none>
I20250111 02:52:08.747133 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:08.747196 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:08.747318 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:08.747921 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "1975f6ffb6f240b389ac8bd499822c7a"
format_stamp: "Formatted at 2025-01-11 02:52:08 on dist-test-slave-4xrj"
I20250111 02:52:08.750276 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:52:08.751597 2670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.751860 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:08.751951 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "1975f6ffb6f240b389ac8bd499822c7a"
format_stamp: "Formatted at 2025-01-11 02:52:08 on dist-test-slave-4xrj"
I20250111 02:52:08.752068 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:08.771512 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:08.772125 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:08.772646 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:08.773416 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:08.773468 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.773538 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:08.773572 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.784811 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:33511
I20250111 02:52:08.784844 2732 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:33511 every 8 connection(s)
I20250111 02:52:08.786324 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:08.789736 2737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:08.790643 2738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:08.791414 2740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:08.792320 2733 heartbeater.cc:346] Connected to a master server at 127.0.103.62:44347
I20250111 02:52:08.792449 2733 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:08.792483 412 server_base.cc:1034] running on GCE node
I20250111 02:52:08.792722 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:08.792774 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:08.792793 2733 heartbeater.cc:510] Master 127.0.103.62:44347 requested a full tablet report, sending...
I20250111 02:52:08.792820 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563928792817 us; error 0 us; skew 500 ppm
I20250111 02:52:08.792946 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:08.793680 2606 ts_manager.cc:194] Registered new tserver with Master: 1975f6ffb6f240b389ac8bd499822c7a (127.0.103.1:33511)
I20250111 02:52:08.794298 412 webserver.cc:458] Webserver started at http://127.0.103.2:37415/ using document root <none> and password file <none>
I20250111 02:52:08.794545 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:08.794617 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:08.794716 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:08.795130 2606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60510
I20250111 02:52:08.795424 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "7f839112be404fcb864cad7f52a39a2d"
format_stamp: "Formatted at 2025-01-11 02:52:08 on dist-test-slave-4xrj"
I20250111 02:52:08.797945 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:08.799232 2745 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.799499 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:08.799587 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "7f839112be404fcb864cad7f52a39a2d"
format_stamp: "Formatted at 2025-01-11 02:52:08 on dist-test-slave-4xrj"
I20250111 02:52:08.799677 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:08.812948 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:08.813437 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:08.813951 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:08.814663 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:08.814718 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.814781 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:08.814816 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.826977 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:38463
I20250111 02:52:08.827003 2807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:38463 every 8 connection(s)
I20250111 02:52:08.828424 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:08.831866 2811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:08.832769 2812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:08.833735 2814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:08.834417 2808 heartbeater.cc:346] Connected to a master server at 127.0.103.62:44347
I20250111 02:52:08.834523 2808 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:08.834542 412 server_base.cc:1034] running on GCE node
I20250111 02:52:08.834777 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:08.834833 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:08.834861 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563928834857 us; error 0 us; skew 500 ppm
I20250111 02:52:08.834900 2808 heartbeater.cc:510] Master 127.0.103.62:44347 requested a full tablet report, sending...
I20250111 02:52:08.834992 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:08.835769 2606 ts_manager.cc:194] Registered new tserver with Master: 7f839112be404fcb864cad7f52a39a2d (127.0.103.2:38463)
I20250111 02:52:08.836267 412 webserver.cc:458] Webserver started at http://127.0.103.3:36297/ using document root <none> and password file <none>
I20250111 02:52:08.836510 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:08.836580 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:08.836683 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:08.836802 2606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60514
I20250111 02:52:08.837347 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "90969aeaf5d449f792e17b2543946727"
format_stamp: "Formatted at 2025-01-11 02:52:08 on dist-test-slave-4xrj"
I20250111 02:52:08.839560 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:52:08.840974 2819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.841287 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:52:08.841382 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "90969aeaf5d449f792e17b2543946727"
format_stamp: "Formatted at 2025-01-11 02:52:08 on dist-test-slave-4xrj"
I20250111 02:52:08.841471 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:08.858875 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:08.859375 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:08.859884 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:08.860615 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:08.860669 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.860735 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:08.860770 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:08.872963 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:36381
I20250111 02:52:08.872987 2881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:36381 every 8 connection(s)
I20250111 02:52:08.878697 2882 heartbeater.cc:346] Connected to a master server at 127.0.103.62:44347
I20250111 02:52:08.878798 2882 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:08.879087 2882 heartbeater.cc:510] Master 127.0.103.62:44347 requested a full tablet report, sending...
I20250111 02:52:08.879819 2606 ts_manager.cc:194] Registered new tserver with Master: 90969aeaf5d449f792e17b2543946727 (127.0.103.3:36381)
I20250111 02:52:08.880463 2606 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60518
I20250111 02:52:08.880571 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006732211s
I20250111 02:52:09.796664 2733 heartbeater.cc:502] Master 127.0.103.62:44347 was elected leader, sending a full tablet report...
I20250111 02:52:09.837949 2808 heartbeater.cc:502] Master 127.0.103.62:44347 was elected leader, sending a full tablet report...
I20250111 02:52:09.881569 2882 heartbeater.cc:502] Master 127.0.103.62:44347 was elected leader, sending a full tablet report...
I20250111 02:52:10.706745 2660 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:12.707055 2660 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:14.707419 2660 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:16.707811 2660 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:18.708187 2660 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:20.708576 2660 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:20.968348 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:52:20.980789 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:20.996320 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:52:20.998729 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:52:21.006721 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:21.018946 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:52:21.021306 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:52:21.029431 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:21.041792 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:52:21.044035 412 master.cc:537] Master@127.0.103.62:44347 shutting down...
I20250111 02:52:21.050868 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:52:21.051074 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:21.051180 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 28b10236cd074f7a8c3252a5890bff98: stopping tablet replica
I20250111 02:52:21.063920 412 master.cc:559] Master@127.0.103.62:44347 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12424 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20250111 02:52:21.081450 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:34261
I20250111 02:52:21.081902 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:21.084774 2886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:21.085918 2887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:21.086678 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:21.087474 412 server_base.cc:1034] running on GCE node
I20250111 02:52:21.087687 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:21.087731 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:21.087764 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563941087760 us; error 0 us; skew 500 ppm
I20250111 02:52:21.087893 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:21.088922 412 webserver.cc:458] Webserver started at http://127.0.103.62:40133/ using document root <none> and password file <none>
I20250111 02:52:21.089104 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:21.089162 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:21.089274 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:21.089809 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "d4d61caba042456aa1a81a93af0f7ad4"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.091972 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:52:21.093235 2894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.093523 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:21.093608 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "d4d61caba042456aa1a81a93af0f7ad4"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.093700 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:21.103463 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:21.103880 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:21.112408 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:34261
I20250111 02:52:21.112429 2945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:34261 every 8 connection(s)
I20250111 02:52:21.113409 2946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:21.116672 2946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4: Bootstrap starting.
I20250111 02:52:21.117552 2946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:21.118732 2946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4: No bootstrap required, opened a new log
I20250111 02:52:21.119186 2946 raft_consensus.cc:357] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d61caba042456aa1a81a93af0f7ad4" member_type: VOTER }
I20250111 02:52:21.119290 2946 raft_consensus.cc:383] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:21.119338 2946 raft_consensus.cc:738] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d61caba042456aa1a81a93af0f7ad4, State: Initialized, Role: FOLLOWER
I20250111 02:52:21.119463 2946 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d61caba042456aa1a81a93af0f7ad4" member_type: VOTER }
I20250111 02:52:21.119541 2946 raft_consensus.cc:397] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:52:21.119575 2946 raft_consensus.cc:491] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:52:21.119617 2946 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:21.121013 2946 raft_consensus.cc:513] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d61caba042456aa1a81a93af0f7ad4" member_type: VOTER }
I20250111 02:52:21.121152 2946 leader_election.cc:304] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4d61caba042456aa1a81a93af0f7ad4; no voters:
I20250111 02:52:21.121497 2946 leader_election.cc:290] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:52:21.121589 2949 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:21.122015 2949 raft_consensus.cc:695] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 1 LEADER]: Becoming Leader. State: Replica: d4d61caba042456aa1a81a93af0f7ad4, State: Running, Role: LEADER
I20250111 02:52:21.122257 2946 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:52:21.122226 2949 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d61caba042456aa1a81a93af0f7ad4" member_type: VOTER }
I20250111 02:52:21.122960 2951 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d61caba042456aa1a81a93af0f7ad4. Latest consensus state: current_term: 1 leader_uuid: "d4d61caba042456aa1a81a93af0f7ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d61caba042456aa1a81a93af0f7ad4" member_type: VOTER } }
I20250111 02:52:21.123174 2951 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [sys.catalog]: This master's current role is: LEADER
I20250111 02:52:21.123171 2950 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4d61caba042456aa1a81a93af0f7ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d61caba042456aa1a81a93af0f7ad4" member_type: VOTER } }
I20250111 02:52:21.123378 2950 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [sys.catalog]: This master's current role is: LEADER
I20250111 02:52:21.123977 2954 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:52:21.125211 2954 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:52:21.126996 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:52:21.127696 2954 catalog_manager.cc:1348] Generated new cluster ID: 2d1bb7cc2fd0458b909520ba301c2e3b
I20250111 02:52:21.127766 2954 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:52:21.143792 2954 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:52:21.144433 2954 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:52:21.154170 2954 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4: Generated new TSK 0
I20250111 02:52:21.154328 2954 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:52:21.159691 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:21.162772 2967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:21.163446 2968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:21.164180 2970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:21.165313 412 server_base.cc:1034] running on GCE node
I20250111 02:52:21.165536 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:21.165587 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:21.165627 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563941165623 us; error 0 us; skew 500 ppm
I20250111 02:52:21.165781 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:21.166889 412 webserver.cc:458] Webserver started at http://127.0.103.1:39085/ using document root <none> and password file <none>
I20250111 02:52:21.167084 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:21.167145 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:21.167266 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:21.167824 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "36f844239ff841ba8619dcc01bd3f5c1"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.170084 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:21.171403 2975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.171708 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:21.171787 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "36f844239ff841ba8619dcc01bd3f5c1"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.171881 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:21.182525 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:21.182965 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:21.183420 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:21.184166 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:21.184219 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.184281 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:21.184316 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.194407 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:39833
I20250111 02:52:21.194424 3037 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:39833 every 8 connection(s)
I20250111 02:52:21.195894 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:21.199018 3042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:21.200124 3043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:21.200954 3045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:21.201398 3038 heartbeater.cc:346] Connected to a master server at 127.0.103.62:34261
I20250111 02:52:21.201486 3038 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:21.201702 412 server_base.cc:1034] running on GCE node
I20250111 02:52:21.201798 3038 heartbeater.cc:510] Master 127.0.103.62:34261 requested a full tablet report, sending...
I20250111 02:52:21.201990 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:21.202045 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:21.202082 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563941202079 us; error 0 us; skew 500 ppm
I20250111 02:52:21.202217 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:21.202611 2911 ts_manager.cc:194] Registered new tserver with Master: 36f844239ff841ba8619dcc01bd3f5c1 (127.0.103.1:39833)
I20250111 02:52:21.203356 412 webserver.cc:458] Webserver started at http://127.0.103.2:36323/ using document root <none> and password file <none>
I20250111 02:52:21.203531 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:21.203595 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:21.203703 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:21.203833 2911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39278
I20250111 02:52:21.204344 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "8f03f32d2aeb4baba9b489dab244f1f3"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.206552 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250111 02:52:21.207811 3050 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.208070 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:21.208159 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "8f03f32d2aeb4baba9b489dab244f1f3"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.208251 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:21.217065 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:21.217466 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:21.217885 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:21.218551 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:21.218602 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.218662 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:21.218701 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.229681 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:33521
I20250111 02:52:21.229703 3112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:33521 every 8 connection(s)
I20250111 02:52:21.231102 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:21.234431 3116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:21.235193 3117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:21.235942 3119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:21.236657 412 server_base.cc:1034] running on GCE node
I20250111 02:52:21.236909 3113 heartbeater.cc:346] Connected to a master server at 127.0.103.62:34261
I20250111 02:52:21.236914 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:21.237001 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:21.237021 3113 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:21.237044 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563941237041 us; error 0 us; skew 500 ppm
I20250111 02:52:21.237174 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:21.237329 3113 heartbeater.cc:510] Master 127.0.103.62:34261 requested a full tablet report, sending...
I20250111 02:52:21.238129 2911 ts_manager.cc:194] Registered new tserver with Master: 8f03f32d2aeb4baba9b489dab244f1f3 (127.0.103.2:33521)
I20250111 02:52:21.238410 412 webserver.cc:458] Webserver started at http://127.0.103.3:32807/ using document root <none> and password file <none>
I20250111 02:52:21.238597 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:21.238662 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:21.238754 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:21.239034 2911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39286
I20250111 02:52:21.239357 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "13b5f276bd744eadbb67d4681c270986"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.241282 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:21.242542 3124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.242802 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:21.242874 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "13b5f276bd744eadbb67d4681c270986"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.242956 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:21.250375 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:21.250808 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:21.251245 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:21.251911 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:21.251961 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.252015 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:21.252045 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.262681 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:35197
I20250111 02:52:21.262702 3186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:35197 every 8 connection(s)
I20250111 02:52:21.263995 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:21.267223 3190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:21.268271 412 server_base.cc:1034] running on GCE node
W20250111 02:52:21.269034 3191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:21.269544 3187 heartbeater.cc:346] Connected to a master server at 127.0.103.62:34261
I20250111 02:52:21.269645 3187 heartbeater.cc:463] Registering TS with master...
W20250111 02:52:21.269819 3193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:21.269976 3187 heartbeater.cc:510] Master 127.0.103.62:34261 requested a full tablet report, sending...
I20250111 02:52:21.270200 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:21.270262 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:21.270292 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563941270289 us; error 0 us; skew 500 ppm
I20250111 02:52:21.270424 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:21.270758 2911 ts_manager.cc:194] Registered new tserver with Master: 13b5f276bd744eadbb67d4681c270986 (127.0.103.3:35197)
I20250111 02:52:21.271590 412 webserver.cc:458] Webserver started at http://127.0.103.4:40313/ using document root <none> and password file <none>
I20250111 02:52:21.271703 2911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39290
I20250111 02:52:21.271791 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:21.271852 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:21.271958 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:21.272534 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-3-root/instance:
uuid: "aa04fe03bcfa4613b4630aaa5de307b9"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.274654 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:52:21.275985 3198 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.276262 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:21.276345 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-3-root
uuid: "aa04fe03bcfa4613b4630aaa5de307b9"
format_stamp: "Formatted at 2025-01-11 02:52:21 on dist-test-slave-4xrj"
I20250111 02:52:21.276427 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1736563872491422-412-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:21.289330 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:21.289764 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:21.290233 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:21.290927 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:21.290980 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.291040 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:21.291074 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:21.301457 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.4:36641
I20250111 02:52:21.301481 3260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.4:36641 every 8 connection(s)
I20250111 02:52:21.307230 3261 heartbeater.cc:346] Connected to a master server at 127.0.103.62:34261
I20250111 02:52:21.307339 3261 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:21.307655 3261 heartbeater.cc:510] Master 127.0.103.62:34261 requested a full tablet report, sending...
I20250111 02:52:21.308320 2911 ts_manager.cc:194] Registered new tserver with Master: aa04fe03bcfa4613b4630aaa5de307b9 (127.0.103.4:36641)
I20250111 02:52:21.308967 412 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.006723579s
I20250111 02:52:21.308992 2911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39298
I20250111 02:52:22.204963 3038 heartbeater.cc:502] Master 127.0.103.62:34261 was elected leader, sending a full tablet report...
I20250111 02:52:22.240123 3113 heartbeater.cc:502] Master 127.0.103.62:34261 was elected leader, sending a full tablet report...
I20250111 02:52:22.272822 3187 heartbeater.cc:502] Master 127.0.103.62:34261 was elected leader, sending a full tablet report...
I20250111 02:52:22.310262 3261 heartbeater.cc:502] Master 127.0.103.62:34261 was elected leader, sending a full tablet report...
I20250111 02:52:22.342633 412 test_util.cc:274] Using random seed: 1290387426
I20250111 02:52:22.352162 2911 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39326:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250111 02:52:22.366096 3152 tablet_service.cc:1467] Processing CreateTablet for tablet fa297b52b731490aa9467bcee6af3301 (DEFAULT_TABLE table=test-workload [id=b34323df7ae54db59a5b54c256b9cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:52:22.366568 3152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa297b52b731490aa9467bcee6af3301. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:22.367765 3226 tablet_service.cc:1467] Processing CreateTablet for tablet fa297b52b731490aa9467bcee6af3301 (DEFAULT_TABLE table=test-workload [id=b34323df7ae54db59a5b54c256b9cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:52:22.368157 3226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa297b52b731490aa9467bcee6af3301. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:22.368273 3003 tablet_service.cc:1467] Processing CreateTablet for tablet fa297b52b731490aa9467bcee6af3301 (DEFAULT_TABLE table=test-workload [id=b34323df7ae54db59a5b54c256b9cac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:52:22.368593 3003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa297b52b731490aa9467bcee6af3301. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:22.372387 3282 tablet_bootstrap.cc:492] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9: Bootstrap starting.
I20250111 02:52:22.373917 3282 tablet_bootstrap.cc:654] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:22.374590 3284 tablet_bootstrap.cc:492] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1: Bootstrap starting.
I20250111 02:52:22.376775 3284 tablet_bootstrap.cc:654] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:22.379480 3284 tablet_bootstrap.cc:492] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1: No bootstrap required, opened a new log
I20250111 02:52:22.379630 3284 ts_tablet_manager.cc:1397] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250111 02:52:22.380533 3284 raft_consensus.cc:357] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.380774 3284 raft_consensus.cc:383] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:22.380842 3284 raft_consensus.cc:738] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f844239ff841ba8619dcc01bd3f5c1, State: Initialized, Role: FOLLOWER
I20250111 02:52:22.380987 3283 tablet_bootstrap.cc:492] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986: Bootstrap starting.
I20250111 02:52:22.381146 3284 consensus_queue.cc:260] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.382346 3284 ts_tablet_manager.cc:1428] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250111 02:52:22.382822 3282 tablet_bootstrap.cc:492] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9: No bootstrap required, opened a new log
I20250111 02:52:22.382930 3282 ts_tablet_manager.cc:1397] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250111 02:52:22.383222 3283 tablet_bootstrap.cc:654] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:22.383558 3282 raft_consensus.cc:357] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.383726 3282 raft_consensus.cc:383] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:22.383783 3282 raft_consensus.cc:738] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa04fe03bcfa4613b4630aaa5de307b9, State: Initialized, Role: FOLLOWER
I20250111 02:52:22.383957 3282 consensus_queue.cc:260] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.385005 3282 ts_tablet_manager.cc:1428] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250111 02:52:22.386235 3283 tablet_bootstrap.cc:492] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986: No bootstrap required, opened a new log
I20250111 02:52:22.386363 3283 ts_tablet_manager.cc:1397] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250111 02:52:22.386870 3283 raft_consensus.cc:357] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.387008 3283 raft_consensus.cc:383] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:22.387070 3283 raft_consensus.cc:738] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b5f276bd744eadbb67d4681c270986, State: Initialized, Role: FOLLOWER
I20250111 02:52:22.387231 3283 consensus_queue.cc:260] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.387836 3283 ts_tablet_manager.cc:1428] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20250111 02:52:22.446139 3039 tablet.cc:2367] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:52:22.460487 3287 raft_consensus.cc:491] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:22.460652 3287 raft_consensus.cc:513] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.461349 3287 leader_election.cc:290] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13b5f276bd744eadbb67d4681c270986 (127.0.103.3:35197), aa04fe03bcfa4613b4630aaa5de307b9 (127.0.103.4:36641)
I20250111 02:52:22.468570 3236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa297b52b731490aa9467bcee6af3301" candidate_uuid: "36f844239ff841ba8619dcc01bd3f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" is_pre_election: true
I20250111 02:52:22.468570 3162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa297b52b731490aa9467bcee6af3301" candidate_uuid: "36f844239ff841ba8619dcc01bd3f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b5f276bd744eadbb67d4681c270986" is_pre_election: true
I20250111 02:52:22.468820 3162 raft_consensus.cc:2463] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f844239ff841ba8619dcc01bd3f5c1 in term 0.
I20250111 02:52:22.468838 3236 raft_consensus.cc:2463] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f844239ff841ba8619dcc01bd3f5c1 in term 0.
I20250111 02:52:22.469246 2979 leader_election.cc:304] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f844239ff841ba8619dcc01bd3f5c1, aa04fe03bcfa4613b4630aaa5de307b9; no voters:
I20250111 02:52:22.469482 3287 raft_consensus.cc:2798] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:22.469580 3287 raft_consensus.cc:491] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:22.469671 3287 raft_consensus.cc:3054] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:22.471261 3287 raft_consensus.cc:513] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.471724 3287 leader_election.cc:290] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [CANDIDATE]: Term 1 election: Requested vote from peers 13b5f276bd744eadbb67d4681c270986 (127.0.103.3:35197), aa04fe03bcfa4613b4630aaa5de307b9 (127.0.103.4:36641)
I20250111 02:52:22.472080 3162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa297b52b731490aa9467bcee6af3301" candidate_uuid: "36f844239ff841ba8619dcc01bd3f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b5f276bd744eadbb67d4681c270986"
I20250111 02:52:22.472146 3236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa297b52b731490aa9467bcee6af3301" candidate_uuid: "36f844239ff841ba8619dcc01bd3f5c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa04fe03bcfa4613b4630aaa5de307b9"
I20250111 02:52:22.472224 3162 raft_consensus.cc:3054] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:22.472348 3236 raft_consensus.cc:3054] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:22.474006 3162 raft_consensus.cc:2463] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f844239ff841ba8619dcc01bd3f5c1 in term 1.
I20250111 02:52:22.474107 3236 raft_consensus.cc:2463] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f844239ff841ba8619dcc01bd3f5c1 in term 1.
I20250111 02:52:22.474458 2979 leader_election.cc:304] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b5f276bd744eadbb67d4681c270986, 36f844239ff841ba8619dcc01bd3f5c1; no voters:
I20250111 02:52:22.474668 3287 raft_consensus.cc:2798] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:22.475093 3287 raft_consensus.cc:695] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 1 LEADER]: Becoming Leader. State: Replica: 36f844239ff841ba8619dcc01bd3f5c1, State: Running, Role: LEADER
I20250111 02:52:22.475338 3287 consensus_queue.cc:237] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } }
I20250111 02:52:22.477341 2910 catalog_manager.cc:5526] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f844239ff841ba8619dcc01bd3f5c1 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "36f844239ff841ba8619dcc01bd3f5c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36f844239ff841ba8619dcc01bd3f5c1" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39833 } health_report { overall_health: HEALTHY } } }
I20250111 02:52:22.860249 3287 consensus_queue.cc:1035] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa04fe03bcfa4613b4630aaa5de307b9" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:52:22.865181 3293 consensus_queue.cc:1035] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b5f276bd744eadbb67d4681c270986" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 35197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:52:23.126988 2964 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:23.127420 2964 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:23.127537 2964 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:25.127913 2964 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:25.128350 2964 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:25.128444 2964 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:27.128815 2964 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:27.129215 2964 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:27.129305 2964 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:29.129634 2964 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:29.130065 2964 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:29.130159 2964 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:31.130537 2964 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:31.130966 2964 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:31.131057 2964 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:33.131472 2964 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:33.131925 2964 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:33.132030 2964 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:33.534965 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:52:33.546046 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:33.546346 412 tablet_replica.cc:331] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1: stopping tablet replica
I20250111 02:52:33.546543 412 raft_consensus.cc:2238] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:52:33.546860 412 raft_consensus.cc:2267] T fa297b52b731490aa9467bcee6af3301 P 36f844239ff841ba8619dcc01bd3f5c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:33.561966 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:52:33.573163 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:52:33.581130 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:33.593710 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:52:33.596045 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:52:33.602792 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:33.603019 412 tablet_replica.cc:331] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986: stopping tablet replica
I20250111 02:52:33.603158 412 raft_consensus.cc:2238] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:33.603300 412 raft_consensus.cc:2267] T fa297b52b731490aa9467bcee6af3301 P 13b5f276bd744eadbb67d4681c270986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:33.616186 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:52:33.618660 412 tablet_server.cc:178] TabletServer@127.0.103.4:0 shutting down...
I20250111 02:52:33.625614 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:33.625823 412 tablet_replica.cc:331] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9: stopping tablet replica
I20250111 02:52:33.626010 412 raft_consensus.cc:2238] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:33.626180 412 raft_consensus.cc:2267] T fa297b52b731490aa9467bcee6af3301 P aa04fe03bcfa4613b4630aaa5de307b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:33.639045 412 tablet_server.cc:195] TabletServer@127.0.103.4:0 shutdown complete.
I20250111 02:52:33.641439 412 master.cc:537] Master@127.0.103.62:34261 shutting down...
I20250111 02:52:33.648576 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:52:33.648777 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:33.648883 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P d4d61caba042456aa1a81a93af0f7ad4: stopping tablet replica
I20250111 02:52:33.662114 412 master.cc:559] Master@127.0.103.62:34261 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12590 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20250111 02:52:33.672541 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:38735
I20250111 02:52:33.672967 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:33.675971 3300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:33.677268 3301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:33.678236 3303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:33.679111 412 server_base.cc:1034] running on GCE node
I20250111 02:52:33.679299 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:33.679345 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:33.679379 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563953679376 us; error 0 us; skew 500 ppm
I20250111 02:52:33.679507 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:33.686167 412 webserver.cc:458] Webserver started at http://127.0.103.62:37387/ using document root <none> and password file <none>
I20250111 02:52:33.686378 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:33.686435 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:33.686566 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:33.687163 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "317275ab1a584af89c7f50ca8741b858"
format_stamp: "Formatted at 2025-01-11 02:52:33 on dist-test-slave-4xrj"
I20250111 02:52:33.689359 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:33.690940 3308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.691239 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:33.691327 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "317275ab1a584af89c7f50ca8741b858"
format_stamp: "Formatted at 2025-01-11 02:52:33 on dist-test-slave-4xrj"
I20250111 02:52:33.691428 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:33.712332 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:33.712781 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:33.721988 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:38735
I20250111 02:52:33.722007 3359 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:38735 every 8 connection(s)
I20250111 02:52:33.723050 3360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:33.726478 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858: Bootstrap starting.
I20250111 02:52:33.727515 3360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:33.728650 3360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858: No bootstrap required, opened a new log
I20250111 02:52:33.729166 3360 raft_consensus.cc:357] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317275ab1a584af89c7f50ca8741b858" member_type: VOTER }
I20250111 02:52:33.729286 3360 raft_consensus.cc:383] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:33.729322 3360 raft_consensus.cc:738] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 317275ab1a584af89c7f50ca8741b858, State: Initialized, Role: FOLLOWER
I20250111 02:52:33.729455 3360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317275ab1a584af89c7f50ca8741b858" member_type: VOTER }
I20250111 02:52:33.729560 3360 raft_consensus.cc:397] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:52:33.729597 3360 raft_consensus.cc:491] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:52:33.729642 3360 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:33.731112 3360 raft_consensus.cc:513] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317275ab1a584af89c7f50ca8741b858" member_type: VOTER }
I20250111 02:52:33.731287 3360 leader_election.cc:304] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 317275ab1a584af89c7f50ca8741b858; no voters:
I20250111 02:52:33.731616 3360 leader_election.cc:290] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:52:33.731729 3363 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:33.732162 3363 raft_consensus.cc:695] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 1 LEADER]: Becoming Leader. State: Replica: 317275ab1a584af89c7f50ca8741b858, State: Running, Role: LEADER
I20250111 02:52:33.732357 3363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317275ab1a584af89c7f50ca8741b858" member_type: VOTER }
I20250111 02:52:33.732404 3360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:52:33.733058 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 317275ab1a584af89c7f50ca8741b858. Latest consensus state: current_term: 1 leader_uuid: "317275ab1a584af89c7f50ca8741b858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317275ab1a584af89c7f50ca8741b858" member_type: VOTER } }
I20250111 02:52:33.733227 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [sys.catalog]: This master's current role is: LEADER
I20250111 02:52:33.733196 3364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "317275ab1a584af89c7f50ca8741b858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "317275ab1a584af89c7f50ca8741b858" member_type: VOTER } }
I20250111 02:52:33.733306 3364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [sys.catalog]: This master's current role is: LEADER
I20250111 02:52:33.734122 3367 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:52:33.735587 3367 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:52:33.737264 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:52:33.738210 3367 catalog_manager.cc:1348] Generated new cluster ID: 92c57bcb622849d99fe030043041b425
I20250111 02:52:33.738281 3367 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:52:33.768968 3367 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:52:33.769726 3367 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:52:33.775058 3367 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858: Generated new TSK 0
I20250111 02:52:33.775249 3367 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:52:33.802007 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:33.805255 3381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:33.806334 3382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:33.807308 3384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:33.808125 412 server_base.cc:1034] running on GCE node
I20250111 02:52:33.808310 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:33.808360 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:33.808401 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563953808397 us; error 0 us; skew 500 ppm
I20250111 02:52:33.808532 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:33.809650 412 webserver.cc:458] Webserver started at http://127.0.103.1:33549/ using document root <none> and password file <none>
I20250111 02:52:33.809875 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:33.809948 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:33.810079 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:33.810679 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "6fdab1556adf49999c693c61eb0a2256"
format_stamp: "Formatted at 2025-01-11 02:52:33 on dist-test-slave-4xrj"
I20250111 02:52:33.813165 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:33.814723 3389 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.815029 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:52:33.815136 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "6fdab1556adf49999c693c61eb0a2256"
format_stamp: "Formatted at 2025-01-11 02:52:33 on dist-test-slave-4xrj"
I20250111 02:52:33.815236 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:33.823568 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:33.824007 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:33.824474 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:33.825289 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:33.825346 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.825407 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:33.825440 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.837049 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:39981
I20250111 02:52:33.837082 3451 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:39981 every 8 connection(s)
I20250111 02:52:33.838614 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:33.842046 3456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:33.843075 3457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:33.844058 3459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:33.844808 412 server_base.cc:1034] running on GCE node
I20250111 02:52:33.845027 3452 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38735
I20250111 02:52:33.845139 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250111 02:52:33.845168 3452 heartbeater.cc:463] Registering TS with master...
W20250111 02:52:33.845196 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:33.845234 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563953845231 us; error 0 us; skew 500 ppm
I20250111 02:52:33.845358 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:33.845537 3452 heartbeater.cc:510] Master 127.0.103.62:38735 requested a full tablet report, sending...
I20250111 02:52:33.846501 3325 ts_manager.cc:194] Registered new tserver with Master: 6fdab1556adf49999c693c61eb0a2256 (127.0.103.1:39981)
I20250111 02:52:33.846716 412 webserver.cc:458] Webserver started at http://127.0.103.2:46521/ using document root <none> and password file <none>
I20250111 02:52:33.846922 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:33.847000 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:33.847127 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:33.847834 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "ba6150537b6a46b0b7c81b9b4322a97b"
format_stamp: "Formatted at 2025-01-11 02:52:33 on dist-test-slave-4xrj"
I20250111 02:52:33.847921 3325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46662
I20250111 02:52:33.850353 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:52:33.851681 3464 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.851969 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:52:33.852046 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "ba6150537b6a46b0b7c81b9b4322a97b"
format_stamp: "Formatted at 2025-01-11 02:52:33 on dist-test-slave-4xrj"
I20250111 02:52:33.852151 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:33.871596 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:33.872042 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:33.872519 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:33.873241 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:33.873296 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.873361 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:33.873396 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.884330 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:34811
I20250111 02:52:33.884364 3526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:34811 every 8 connection(s)
I20250111 02:52:33.885634 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:33.889124 3530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:33.889842 3531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:33.890676 412 server_base.cc:1034] running on GCE node
W20250111 02:52:33.891436 3533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:33.891641 3527 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38735
I20250111 02:52:33.891773 3527 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:33.891831 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:33.891891 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:33.891927 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563953891923 us; error 0 us; skew 500 ppm
I20250111 02:52:33.892120 3527 heartbeater.cc:510] Master 127.0.103.62:38735 requested a full tablet report, sending...
I20250111 02:52:33.892943 3325 ts_manager.cc:194] Registered new tserver with Master: ba6150537b6a46b0b7c81b9b4322a97b (127.0.103.2:34811)
I20250111 02:52:33.893782 3325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46676
I20250111 02:52:33.897747 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:33.898857 412 webserver.cc:458] Webserver started at http://127.0.103.3:39259/ using document root <none> and password file <none>
I20250111 02:52:33.899012 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:33.899060 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:33.899147 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:33.899600 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10"
format_stamp: "Formatted at 2025-01-11 02:52:33 on dist-test-slave-4xrj"
I20250111 02:52:33.901472 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250111 02:52:33.902766 3538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.903024 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:52:33.903103 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10"
format_stamp: "Formatted at 2025-01-11 02:52:33 on dist-test-slave-4xrj"
I20250111 02:52:33.903198 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:33.919045 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:33.919450 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:33.919900 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:33.920555 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:33.920603 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.920661 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:33.920699 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:33.930447 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:45465
I20250111 02:52:33.930471 3600 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:45465 every 8 connection(s)
I20250111 02:52:33.935900 3601 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38735
I20250111 02:52:33.936002 3601 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:33.936278 3601 heartbeater.cc:510] Master 127.0.103.62:38735 requested a full tablet report, sending...
I20250111 02:52:33.936980 3325 ts_manager.cc:194] Registered new tserver with Master: 3735bd28a8fe49c2ac7c2e706f9f6d10 (127.0.103.3:45465)
I20250111 02:52:33.937613 3325 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46686
I20250111 02:52:33.937824 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006690671s
I20250111 02:52:34.849324 3452 heartbeater.cc:502] Master 127.0.103.62:38735 was elected leader, sending a full tablet report...
I20250111 02:52:34.895202 3527 heartbeater.cc:502] Master 127.0.103.62:38735 was elected leader, sending a full tablet report...
I20250111 02:52:34.938855 3601 heartbeater.cc:502] Master 127.0.103.62:38735 was elected leader, sending a full tablet report...
I20250111 02:52:34.970957 412 test_util.cc:274] Using random seed: 1303015752
I20250111 02:52:34.980507 3325 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47812:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:52:34.981061 3325 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:52:35.000772 3492 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5c5b3b8a944917948f011f0ab875b5 (DEFAULT_TABLE table=test-workload [id=e502bf26a53e4b9683766fa8eafdab3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:52:35.001173 3492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5c5b3b8a944917948f011f0ab875b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:35.007318 3566 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5c5b3b8a944917948f011f0ab875b5 (DEFAULT_TABLE table=test-workload [id=e502bf26a53e4b9683766fa8eafdab3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:52:35.007591 3622 tablet_bootstrap.cc:492] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b: Bootstrap starting.
I20250111 02:52:35.007684 3566 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5c5b3b8a944917948f011f0ab875b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:35.008792 3622 tablet_bootstrap.cc:654] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:35.010462 3622 tablet_bootstrap.cc:492] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b: No bootstrap required, opened a new log
I20250111 02:52:35.010651 3622 ts_tablet_manager.cc:1397] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250111 02:52:35.011205 3622 raft_consensus.cc:357] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.011384 3622 raft_consensus.cc:383] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:35.011452 3622 raft_consensus.cc:738] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba6150537b6a46b0b7c81b9b4322a97b, State: Initialized, Role: FOLLOWER
I20250111 02:52:35.011633 3622 consensus_queue.cc:260] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.011672 3417 tablet_service.cc:1467] Processing CreateTablet for tablet 8c5c5b3b8a944917948f011f0ab875b5 (DEFAULT_TABLE table=test-workload [id=e502bf26a53e4b9683766fa8eafdab3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:52:35.011989 3417 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c5c5b3b8a944917948f011f0ab875b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:35.012347 3622 ts_tablet_manager.cc:1428] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250111 02:52:35.015837 3625 tablet_bootstrap.cc:492] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256: Bootstrap starting.
I20250111 02:52:35.016584 3626 tablet_bootstrap.cc:492] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10: Bootstrap starting.
I20250111 02:52:35.017024 3625 tablet_bootstrap.cc:654] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:35.017788 3626 tablet_bootstrap.cc:654] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:35.018323 3625 tablet_bootstrap.cc:492] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256: No bootstrap required, opened a new log
I20250111 02:52:35.018442 3625 ts_tablet_manager.cc:1397] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:52:35.018954 3625 raft_consensus.cc:357] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.019071 3626 tablet_bootstrap.cc:492] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10: No bootstrap required, opened a new log
I20250111 02:52:35.019107 3625 raft_consensus.cc:383] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:35.019189 3626 ts_tablet_manager.cc:1397] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:52:35.019196 3625 raft_consensus.cc:738] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fdab1556adf49999c693c61eb0a2256, State: Initialized, Role: FOLLOWER
I20250111 02:52:35.019408 3625 consensus_queue.cc:260] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.019812 3626 raft_consensus.cc:357] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.019937 3626 raft_consensus.cc:383] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:35.019999 3626 raft_consensus.cc:738] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3735bd28a8fe49c2ac7c2e706f9f6d10, State: Initialized, Role: FOLLOWER
I20250111 02:52:35.020190 3626 consensus_queue.cc:260] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.020774 3626 ts_tablet_manager.cc:1428] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250111 02:52:35.022197 3625 ts_tablet_manager.cc:1428] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250111 02:52:35.045425 3630 raft_consensus.cc:491] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:35.045557 3630 raft_consensus.cc:513] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.046267 3630 leader_election.cc:290] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba6150537b6a46b0b7c81b9b4322a97b (127.0.103.2:34811), 6fdab1556adf49999c693c61eb0a2256 (127.0.103.1:39981)
I20250111 02:52:35.051292 3427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5c5b3b8a944917948f011f0ab875b5" candidate_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fdab1556adf49999c693c61eb0a2256" is_pre_election: true
I20250111 02:52:35.051384 3502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5c5b3b8a944917948f011f0ab875b5" candidate_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" is_pre_election: true
I20250111 02:52:35.051508 3502 raft_consensus.cc:2463] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3735bd28a8fe49c2ac7c2e706f9f6d10 in term 0.
I20250111 02:52:35.051508 3427 raft_consensus.cc:2463] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3735bd28a8fe49c2ac7c2e706f9f6d10 in term 0.
I20250111 02:52:35.051898 3539 leader_election.cc:304] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3735bd28a8fe49c2ac7c2e706f9f6d10, 6fdab1556adf49999c693c61eb0a2256, ba6150537b6a46b0b7c81b9b4322a97b; no voters:
I20250111 02:52:35.052134 3630 raft_consensus.cc:2798] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:35.052203 3630 raft_consensus.cc:491] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:35.052269 3630 raft_consensus.cc:3054] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:35.053896 3630 raft_consensus.cc:513] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.054375 3630 leader_election.cc:290] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [CANDIDATE]: Term 1 election: Requested vote from peers ba6150537b6a46b0b7c81b9b4322a97b (127.0.103.2:34811), 6fdab1556adf49999c693c61eb0a2256 (127.0.103.1:39981)
I20250111 02:52:35.054688 3502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5c5b3b8a944917948f011f0ab875b5" candidate_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba6150537b6a46b0b7c81b9b4322a97b"
I20250111 02:52:35.054695 3427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c5c5b3b8a944917948f011f0ab875b5" candidate_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fdab1556adf49999c693c61eb0a2256"
I20250111 02:52:35.054833 3427 raft_consensus.cc:3054] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:35.054852 3502 raft_consensus.cc:3054] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:35.056370 3502 raft_consensus.cc:2463] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3735bd28a8fe49c2ac7c2e706f9f6d10 in term 1.
I20250111 02:52:35.056383 3427 raft_consensus.cc:2463] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3735bd28a8fe49c2ac7c2e706f9f6d10 in term 1.
I20250111 02:52:35.056676 3542 leader_election.cc:304] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3735bd28a8fe49c2ac7c2e706f9f6d10, ba6150537b6a46b0b7c81b9b4322a97b; no voters:
I20250111 02:52:35.056906 3630 raft_consensus.cc:2798] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:35.057263 3630 raft_consensus.cc:695] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 1 LEADER]: Becoming Leader. State: Replica: 3735bd28a8fe49c2ac7c2e706f9f6d10, State: Running, Role: LEADER
I20250111 02:52:35.057462 3630 consensus_queue.cc:237] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } }
I20250111 02:52:35.059932 3324 catalog_manager.cc:5526] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3735bd28a8fe49c2ac7c2e706f9f6d10 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3735bd28a8fe49c2ac7c2e706f9f6d10" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 45465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:35.458307 3630 consensus_queue.cc:1035] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fdab1556adf49999c693c61eb0a2256" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:52:35.462333 3630 consensus_queue.cc:1035] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba6150537b6a46b0b7c81b9b4322a97b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:52:35.736753 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:35.737219 3378 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:35.737655 3378 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:35.737758 3378 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:36.739554 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:37.738106 3378 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:37.738503 3378 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:37.738579 3378 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:37.741658 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:38.743757 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:39.738881 3378 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:39.739270 3378 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:39.739347 3378 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:39.745898 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:40.748123 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:41.739650 3378 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:41.740064 3378 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:41.740164 3378 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:41.750336 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:42.752571 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:43.740510 3378 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:43.740909 3378 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:43.740983 3378 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:43.754853 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:44.757019 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:45.741323 3378 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:45.741709 3378 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:45.741782 3378 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:45.759218 3373 placement_policy_util.cc:407] tablet 8c5c5b3b8a944917948f011f0ab875b5: detected majority of replicas (2 of 3) at location L0
I20250111 02:52:46.119879 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:52:46.135954 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:46.136237 412 tablet_replica.cc:331] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256: stopping tablet replica
I20250111 02:52:46.136431 412 raft_consensus.cc:2238] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:46.136600 412 raft_consensus.cc:2267] T 8c5c5b3b8a944917948f011f0ab875b5 P 6fdab1556adf49999c693c61eb0a2256 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:46.150028 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
W20250111 02:52:46.150008 3539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39981: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20250111 02:52:46.151294 3539 consensus_peers.cc:487] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 -> Peer 6fdab1556adf49999c693c61eb0a2256 (127.0.103.1:39981): Couldn't send request to peer 6fdab1556adf49999c693c61eb0a2256. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:46.152911 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:52:46.160606 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:46.160854 412 tablet_replica.cc:331] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b: stopping tablet replica
I20250111 02:52:46.161010 412 raft_consensus.cc:2238] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:46.161172 412 raft_consensus.cc:2267] T 8c5c5b3b8a944917948f011f0ab875b5 P ba6150537b6a46b0b7c81b9b4322a97b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:46.174269 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:52:46.176836 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:52:46.183984 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:52:46.184222 412 tablet_replica.cc:331] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10: stopping tablet replica
I20250111 02:52:46.184383 412 raft_consensus.cc:2238] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:52:46.184676 412 raft_consensus.cc:2267] T 8c5c5b3b8a944917948f011f0ab875b5 P 3735bd28a8fe49c2ac7c2e706f9f6d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:46.197983 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:52:46.200628 412 master.cc:537] Master@127.0.103.62:38735 shutting down...
I20250111 02:52:46.207567 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:52:46.207756 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:46.207851 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 317275ab1a584af89c7f50ca8741b858: stopping tablet replica
I20250111 02:52:46.228644 412 master.cc:559] Master@127.0.103.62:38735 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12565 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20250111 02:52:46.238597 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:35787
I20250111 02:52:46.239034 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:46.242194 3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:46.243124 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:46.244184 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:46.245158 412 server_base.cc:1034] running on GCE node
I20250111 02:52:46.245388 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:46.245442 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:46.245486 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563966245481 us; error 0 us; skew 500 ppm
I20250111 02:52:46.245626 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:46.251135 412 webserver.cc:458] Webserver started at http://127.0.103.62:43227/ using document root <none> and password file <none>
I20250111 02:52:46.251329 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:46.251395 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:46.251505 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:46.252085 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "edd01f073a6441f39ace89b69fab87ab"
format_stamp: "Formatted at 2025-01-11 02:52:46 on dist-test-slave-4xrj"
I20250111 02:52:46.254390 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:46.255748 3653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.256031 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:46.256122 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "edd01f073a6441f39ace89b69fab87ab"
format_stamp: "Formatted at 2025-01-11 02:52:46 on dist-test-slave-4xrj"
I20250111 02:52:46.256211 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:46.277805 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:46.278312 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:46.288690 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:35787
I20250111 02:52:46.288708 3704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:35787 every 8 connection(s)
I20250111 02:52:46.289893 3705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:46.293313 3705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab: Bootstrap starting.
I20250111 02:52:46.294369 3705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:46.295619 3705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab: No bootstrap required, opened a new log
I20250111 02:52:46.296149 3705 raft_consensus.cc:357] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd01f073a6441f39ace89b69fab87ab" member_type: VOTER }
I20250111 02:52:46.296262 3705 raft_consensus.cc:383] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:46.296325 3705 raft_consensus.cc:738] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd01f073a6441f39ace89b69fab87ab, State: Initialized, Role: FOLLOWER
I20250111 02:52:46.296518 3705 consensus_queue.cc:260] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd01f073a6441f39ace89b69fab87ab" member_type: VOTER }
I20250111 02:52:46.296623 3705 raft_consensus.cc:397] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:52:46.296680 3705 raft_consensus.cc:491] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:52:46.296758 3705 raft_consensus.cc:3054] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:46.298317 3705 raft_consensus.cc:513] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd01f073a6441f39ace89b69fab87ab" member_type: VOTER }
I20250111 02:52:46.298525 3705 leader_election.cc:304] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edd01f073a6441f39ace89b69fab87ab; no voters:
I20250111 02:52:46.298858 3705 leader_election.cc:290] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:52:46.298969 3708 raft_consensus.cc:2798] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:46.299319 3708 raft_consensus.cc:695] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 1 LEADER]: Becoming Leader. State: Replica: edd01f073a6441f39ace89b69fab87ab, State: Running, Role: LEADER
I20250111 02:52:46.299525 3708 consensus_queue.cc:237] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd01f073a6441f39ace89b69fab87ab" member_type: VOTER }
I20250111 02:52:46.299584 3705 sys_catalog.cc:564] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:52:46.300140 3709 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edd01f073a6441f39ace89b69fab87ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd01f073a6441f39ace89b69fab87ab" member_type: VOTER } }
I20250111 02:52:46.300185 3710 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader edd01f073a6441f39ace89b69fab87ab. Latest consensus state: current_term: 1 leader_uuid: "edd01f073a6441f39ace89b69fab87ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd01f073a6441f39ace89b69fab87ab" member_type: VOTER } }
I20250111 02:52:46.300319 3709 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [sys.catalog]: This master's current role is: LEADER
I20250111 02:52:46.300424 3710 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [sys.catalog]: This master's current role is: LEADER
I20250111 02:52:46.301249 3713 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:52:46.302394 3713 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:52:46.304230 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:52:46.306581 3713 catalog_manager.cc:1348] Generated new cluster ID: 9c6be2f74942408d9db02b9c2371f342
I20250111 02:52:46.306649 3713 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:52:46.322916 3713 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:52:46.323602 3713 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:52:46.336128 3713 catalog_manager.cc:5899] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab: Generated new TSK 0
I20250111 02:52:46.336306 3713 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:52:46.338800 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:46.342022 3726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:46.343080 3727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:46.344033 3729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:46.344884 412 server_base.cc:1034] running on GCE node
I20250111 02:52:46.345069 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:46.345121 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:46.345165 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563966345162 us; error 0 us; skew 500 ppm
I20250111 02:52:46.345299 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:46.346504 412 webserver.cc:458] Webserver started at http://127.0.103.1:35443/ using document root <none> and password file <none>
I20250111 02:52:46.346701 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:46.346763 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:46.346891 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:46.347494 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "18a7cd9ed0644218857e3da957957cf8"
format_stamp: "Formatted at 2025-01-11 02:52:46 on dist-test-slave-4xrj"
I20250111 02:52:46.349704 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:46.351117 3734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.351397 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:46.351487 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "18a7cd9ed0644218857e3da957957cf8"
format_stamp: "Formatted at 2025-01-11 02:52:46 on dist-test-slave-4xrj"
I20250111 02:52:46.351580 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:46.378737 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:46.379209 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:46.379650 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:46.380424 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:46.380475 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.380542 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:46.380575 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.390830 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:42435
I20250111 02:52:46.390856 3796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:42435 every 8 connection(s)
I20250111 02:52:46.392237 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:46.395766 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:46.396885 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:46.397836 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:46.398494 3797 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35787
I20250111 02:52:46.398607 3797 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:46.398880 3797 heartbeater.cc:510] Master 127.0.103.62:35787 requested a full tablet report, sending...
I20250111 02:52:46.399322 412 server_base.cc:1034] running on GCE node
I20250111 02:52:46.399559 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:46.399596 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:46.399616 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563966399614 us; error 0 us; skew 500 ppm
I20250111 02:52:46.399674 3670 ts_manager.cc:194] Registered new tserver with Master: 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435)
I20250111 02:52:46.399694 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:46.400674 3670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57638
I20250111 02:52:46.400938 412 webserver.cc:458] Webserver started at http://127.0.103.2:43585/ using document root <none> and password file <none>
I20250111 02:52:46.401182 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:46.401244 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:46.401346 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:46.401930 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "912e2307407d45dd88991584cd0b914d"
format_stamp: "Formatted at 2025-01-11 02:52:46 on dist-test-slave-4xrj"
I20250111 02:52:46.403963 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250111 02:52:46.405321 3809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.405601 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250111 02:52:46.405689 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "912e2307407d45dd88991584cd0b914d"
format_stamp: "Formatted at 2025-01-11 02:52:46 on dist-test-slave-4xrj"
I20250111 02:52:46.405789 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:46.434137 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:46.434615 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:46.435108 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:46.435835 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:46.435890 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.435961 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:46.435998 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.447100 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:38147
I20250111 02:52:46.447129 3871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:38147 every 8 connection(s)
I20250111 02:52:46.448482 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:46.451889 3875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:46.452677 3876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:46.453399 3878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:46.454090 3872 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35787
I20250111 02:52:46.454193 3872 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:46.454520 412 server_base.cc:1034] running on GCE node
I20250111 02:52:46.454546 3872 heartbeater.cc:510] Master 127.0.103.62:35787 requested a full tablet report, sending...
I20250111 02:52:46.454761 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:46.454816 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:46.454859 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563966454854 us; error 0 us; skew 500 ppm
I20250111 02:52:46.455005 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:46.455431 3670 ts_manager.cc:194] Registered new tserver with Master: 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:46.456276 412 webserver.cc:458] Webserver started at http://127.0.103.3:43199/ using document root <none> and password file <none>
I20250111 02:52:46.456341 3670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57642
I20250111 02:52:46.456514 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:46.456593 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:46.456698 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:46.457444 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "4667e917a8b44d0384ced54357111047"
format_stamp: "Formatted at 2025-01-11 02:52:46 on dist-test-slave-4xrj"
I20250111 02:52:46.459635 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:46.460924 3883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.461215 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:46.461301 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "4667e917a8b44d0384ced54357111047"
format_stamp: "Formatted at 2025-01-11 02:52:46 on dist-test-slave-4xrj"
I20250111 02:52:46.461387 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:46.472157 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:46.472572 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:46.473031 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:46.473695 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:46.473747 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.473819 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:46.473879 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:46.484189 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:41021
I20250111 02:52:46.484218 3945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:41021 every 8 connection(s)
I20250111 02:52:46.490139 3946 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35787
I20250111 02:52:46.490232 3946 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:46.490491 3946 heartbeater.cc:510] Master 127.0.103.62:35787 requested a full tablet report, sending...
I20250111 02:52:46.491153 3670 ts_manager.cc:194] Registered new tserver with Master: 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:46.491629 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006692175s
I20250111 02:52:46.491811 3670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57646
I20250111 02:52:47.401860 3797 heartbeater.cc:502] Master 127.0.103.62:35787 was elected leader, sending a full tablet report...
I20250111 02:52:47.457654 3872 heartbeater.cc:502] Master 127.0.103.62:35787 was elected leader, sending a full tablet report...
I20250111 02:52:47.492986 3946 heartbeater.cc:502] Master 127.0.103.62:35787 was elected leader, sending a full tablet report...
I20250111 02:52:47.518940 412 test_util.cc:274] Using random seed: 1315563734
I20250111 02:52:47.531419 3670 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57660:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:52:47.532123 3670 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:52:47.567121 3837 tablet_service.cc:1467] Processing CreateTablet for tablet 1e52445200d144479f643c05ecb3a0f8 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250111 02:52:47.567201 3836 tablet_service.cc:1467] Processing CreateTablet for tablet d08873d252254c8dbda59ce10b6766cb (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250111 02:52:47.567729 3837 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e52445200d144479f643c05ecb3a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.567773 3835 tablet_service.cc:1467] Processing CreateTablet for tablet 358018d07a3d4af0a2102356c90436c0 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250111 02:52:47.568284 3834 tablet_service.cc:1467] Processing CreateTablet for tablet 1be25a13ddfe49af9a73cb6f630f505e (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250111 02:52:47.568465 3833 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf22853b8e742d98610db9dbcfea1c0 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250111 02:52:47.568789 3832 tablet_service.cc:1467] Processing CreateTablet for tablet 6c67b2576c5f45b3b4e291de305ec91c (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250111 02:52:47.569295 3831 tablet_service.cc:1467] Processing CreateTablet for tablet 25a4915f163f405fa38c25cc03ceacc9 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250111 02:52:47.569804 3830 tablet_service.cc:1467] Processing CreateTablet for tablet b3dc432e24e7449c879c2c35ad1559d4 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250111 02:52:47.569989 3828 tablet_service.cc:1467] Processing CreateTablet for tablet e7bedf7b15ad48319668c0000adaf421 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250111 02:52:47.570338 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf60fc472f742518bd4328f9e88b436 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250111 02:52:47.570875 3835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358018d07a3d4af0a2102356c90436c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.570909 3911 tablet_service.cc:1467] Processing CreateTablet for tablet 1e52445200d144479f643c05ecb3a0f8 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250111 02:52:47.571368 3911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e52445200d144479f643c05ecb3a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.572228 3902 tablet_service.cc:1467] Processing CreateTablet for tablet e7bedf7b15ad48319668c0000adaf421 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250111 02:52:47.572604 3902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bedf7b15ad48319668c0000adaf421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.572896 3753 tablet_service.cc:1467] Processing CreateTablet for tablet e7bedf7b15ad48319668c0000adaf421 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250111 02:52:47.573254 3753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bedf7b15ad48319668c0000adaf421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.576948 3762 tablet_service.cc:1467] Processing CreateTablet for tablet 1e52445200d144479f643c05ecb3a0f8 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250111 02:52:47.577239 3762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e52445200d144479f643c05ecb3a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.578307 3903 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf60fc472f742518bd4328f9e88b436 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250111 02:52:47.578824 3903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf60fc472f742518bd4328f9e88b436. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.582264 3761 tablet_service.cc:1467] Processing CreateTablet for tablet d08873d252254c8dbda59ce10b6766cb (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250111 02:52:47.582708 3761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08873d252254c8dbda59ce10b6766cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.586305 3760 tablet_service.cc:1467] Processing CreateTablet for tablet 358018d07a3d4af0a2102356c90436c0 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250111 02:52:47.600097 3760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358018d07a3d4af0a2102356c90436c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.603710 3832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c67b2576c5f45b3b4e291de305ec91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.605929 3902 tablet_service.cc:1467] Processing CreateTablet for tablet e886897718de4e46833c86f6412bdf1a (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250111 02:52:47.608016 3828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bedf7b15ad48319668c0000adaf421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.609411 3833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf22853b8e742d98610db9dbcfea1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.611522 3902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e886897718de4e46833c86f6412bdf1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.612505 3830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dc432e24e7449c879c2c35ad1559d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.612994 3829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf60fc472f742518bd4328f9e88b436. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.614323 3831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a4915f163f405fa38c25cc03ceacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.615053 3910 tablet_service.cc:1467] Processing CreateTablet for tablet d08873d252254c8dbda59ce10b6766cb (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250111 02:52:47.612855 3759 tablet_service.cc:1467] Processing CreateTablet for tablet 1be25a13ddfe49af9a73cb6f630f505e (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250111 02:52:47.615106 3909 tablet_service.cc:1467] Processing CreateTablet for tablet 358018d07a3d4af0a2102356c90436c0 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250111 02:52:47.615677 3836 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08873d252254c8dbda59ce10b6766cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.616531 3834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1be25a13ddfe49af9a73cb6f630f505e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.612310 3758 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf22853b8e742d98610db9dbcfea1c0 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250111 02:52:47.616335 3911 tablet_service.cc:1467] Processing CreateTablet for tablet 0e921916fa0e4883a3028f4fa013bc38 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250111 02:52:47.614279 3756 tablet_service.cc:1467] Processing CreateTablet for tablet 25a4915f163f405fa38c25cc03ceacc9 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250111 02:52:47.613963 3757 tablet_service.cc:1467] Processing CreateTablet for tablet 6c67b2576c5f45b3b4e291de305ec91c (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250111 02:52:47.618801 3966 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.618871 3908 tablet_service.cc:1467] Processing CreateTablet for tablet 1be25a13ddfe49af9a73cb6f630f505e (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250111 02:52:47.619299 3754 tablet_service.cc:1467] Processing CreateTablet for tablet 3bf60fc472f742518bd4328f9e88b436 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250111 02:52:47.620980 3762 tablet_service.cc:1467] Processing CreateTablet for tablet 0e921916fa0e4883a3028f4fa013bc38 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250111 02:52:47.621408 3762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e921916fa0e4883a3028f4fa013bc38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.622653 3759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1be25a13ddfe49af9a73cb6f630f505e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.622915 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c67b2576c5f45b3b4e291de305ec91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.623080 3835 tablet_service.cc:1467] Processing CreateTablet for tablet e886897718de4e46833c86f6412bdf1a (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250111 02:52:47.624261 3911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e921916fa0e4883a3028f4fa013bc38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.627555 3835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e886897718de4e46833c86f6412bdf1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.629662 3909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358018d07a3d4af0a2102356c90436c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.630453 3754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf60fc472f742518bd4328f9e88b436. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.632185 3904 tablet_service.cc:1467] Processing CreateTablet for tablet b3dc432e24e7449c879c2c35ad1559d4 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250111 02:52:47.632277 3755 tablet_service.cc:1467] Processing CreateTablet for tablet b3dc432e24e7449c879c2c35ad1559d4 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250111 02:52:47.633916 3905 tablet_service.cc:1467] Processing CreateTablet for tablet 25a4915f163f405fa38c25cc03ceacc9 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250111 02:52:47.633953 3906 tablet_service.cc:1467] Processing CreateTablet for tablet 6c67b2576c5f45b3b4e291de305ec91c (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250111 02:52:47.634362 3907 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf22853b8e742d98610db9dbcfea1c0 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250111 02:52:47.634881 3966 tablet_bootstrap.cc:654] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.635180 3908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1be25a13ddfe49af9a73cb6f630f505e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.636885 3968 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.637053 3756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a4915f163f405fa38c25cc03ceacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.639320 3904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dc432e24e7449c879c2c35ad1559d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.639525 3905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a4915f163f405fa38c25cc03ceacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.640028 3753 tablet_service.cc:1467] Processing CreateTablet for tablet e886897718de4e46833c86f6412bdf1a (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250111 02:52:47.643510 3967 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.643674 3906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c67b2576c5f45b3b4e291de305ec91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.643893 3907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf22853b8e742d98610db9dbcfea1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.644042 3755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dc432e24e7449c879c2c35ad1559d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.646054 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf22853b8e742d98610db9dbcfea1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.646181 3829 tablet_service.cc:1467] Processing CreateTablet for tablet 0e921916fa0e4883a3028f4fa013bc38 (DEFAULT_TABLE table=test-workload [id=913eeb487ba14c7c9f287fff752a835e]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250111 02:52:47.646517 3829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e921916fa0e4883a3028f4fa013bc38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.649961 3753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e886897718de4e46833c86f6412bdf1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.650354 3910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08873d252254c8dbda59ce10b6766cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:47.684161 3966 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.684355 3966 ts_tablet_manager.cc:1397] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.010s
I20250111 02:52:47.685207 3968 tablet_bootstrap.cc:654] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.685159 3966 raft_consensus.cc:357] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.685393 3966 raft_consensus.cc:383] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.685485 3966 raft_consensus.cc:738] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.685737 3966 consensus_queue.cc:260] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.687065 3966 ts_tablet_manager.cc:1428] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:47.687366 3966 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.688827 3968 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.688985 3968 ts_tablet_manager.cc:1397] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.003s
I20250111 02:52:47.689061 3966 tablet_bootstrap.cc:654] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.689622 3967 tablet_bootstrap.cc:654] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.689709 3968 raft_consensus.cc:357] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.690876 3968 raft_consensus.cc:383] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.690950 3966 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.690969 3968 raft_consensus.cc:738] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.691124 3966 ts_tablet_manager.cc:1397] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250111 02:52:47.691318 3968 consensus_queue.cc:260] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.692092 3968 ts_tablet_manager.cc:1428] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250111 02:52:47.692116 3967 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.692268 3967 ts_tablet_manager.cc:1397] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.050s user 0.000s sys 0.004s
I20250111 02:52:47.692358 3968 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.691857 3966 raft_consensus.cc:357] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.692461 3966 raft_consensus.cc:383] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.692525 3966 raft_consensus.cc:738] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.692801 3966 consensus_queue.cc:260] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.692966 3967 raft_consensus.cc:357] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.693641 3967 raft_consensus.cc:383] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.693753 3967 raft_consensus.cc:738] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.694000 3967 consensus_queue.cc:260] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.694463 3968 tablet_bootstrap.cc:654] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.695465 3966 ts_tablet_manager.cc:1428] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250111 02:52:47.696564 3966 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.698495 3966 tablet_bootstrap.cc:654] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.698933 3967 ts_tablet_manager.cc:1428] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250111 02:52:47.699184 3967 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.700081 3968 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.700234 3968 ts_tablet_manager.cc:1397] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250111 02:52:47.700652 3967 tablet_bootstrap.cc:654] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.701086 3966 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.701231 3966 ts_tablet_manager.cc:1397] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250111 02:52:47.701234 3968 raft_consensus.cc:357] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.701453 3968 raft_consensus.cc:383] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.701521 3968 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.701921 3966 raft_consensus.cc:357] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.702093 3966 raft_consensus.cc:383] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.702163 3966 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.701736 3968 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.702412 3966 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.703881 3966 ts_tablet_manager.cc:1428] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250111 02:52:47.704146 3966 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.704174 3968 ts_tablet_manager.cc:1428] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250111 02:52:47.704392 3968 tablet_bootstrap.cc:492] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.706322 3966 tablet_bootstrap.cc:654] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.706502 3968 tablet_bootstrap.cc:654] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.710435 3968 tablet_bootstrap.cc:492] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.710563 3968 ts_tablet_manager.cc:1397] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250111 02:52:47.710696 3966 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.710815 3966 ts_tablet_manager.cc:1397] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250111 02:52:47.711225 3968 raft_consensus.cc:357] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.711406 3968 raft_consensus.cc:383] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.711349 3966 raft_consensus.cc:357] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.711480 3968 raft_consensus.cc:738] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.711527 3966 raft_consensus.cc:383] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.711597 3966 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.711687 3968 consensus_queue.cc:260] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.711817 3966 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.712211 3968 ts_tablet_manager.cc:1428] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250111 02:52:47.712473 3968 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.712745 3966 ts_tablet_manager.cc:1428] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:52:47.712982 3966 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.714102 3968 tablet_bootstrap.cc:654] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.714660 3966 tablet_bootstrap.cc:654] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.716889 3967 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.717051 3967 ts_tablet_manager.cc:1397] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20250111 02:52:47.717533 3968 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.717692 3968 ts_tablet_manager.cc:1397] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250111 02:52:47.717725 3967 raft_consensus.cc:357] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.717912 3967 raft_consensus.cc:383] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.718003 3967 raft_consensus.cc:738] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.718222 3967 consensus_queue.cc:260] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.718633 3967 ts_tablet_manager.cc:1428] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250111 02:52:47.718699 3968 raft_consensus.cc:357] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.718924 3967 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.718962 3968 raft_consensus.cc:383] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.719048 3968 raft_consensus.cc:738] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.719339 3968 consensus_queue.cc:260] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.720561 3967 tablet_bootstrap.cc:654] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.720626 3968 ts_tablet_manager.cc:1428] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250111 02:52:47.720628 3966 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.720827 3966 ts_tablet_manager.cc:1397] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250111 02:52:47.720927 3968 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.721470 3966 raft_consensus.cc:357] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.721709 3966 raft_consensus.cc:383] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.721778 3966 raft_consensus.cc:738] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.722038 3966 consensus_queue.cc:260] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.722384 3968 tablet_bootstrap.cc:654] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.724678 3966 ts_tablet_manager.cc:1428] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250111 02:52:47.726462 3966 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.729275 3966 tablet_bootstrap.cc:654] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.729817 3967 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.729990 3967 ts_tablet_manager.cc:1397] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20250111 02:52:47.730744 3968 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.730680 3967 raft_consensus.cc:357] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.730858 3967 raft_consensus.cc:383] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.730873 3968 ts_tablet_manager.cc:1397] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250111 02:52:47.730964 3967 raft_consensus.cc:738] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.731228 3967 consensus_queue.cc:260] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.731550 3968 raft_consensus.cc:357] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.731688 3967 ts_tablet_manager.cc:1428] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250111 02:52:47.731738 3968 raft_consensus.cc:383] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.731807 3968 raft_consensus.cc:738] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.731886 3966 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.731967 3967 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.732020 3966 ts_tablet_manager.cc:1397] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250111 02:52:47.732014 3968 consensus_queue.cc:260] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.732549 3968 ts_tablet_manager.cc:1428] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250111 02:52:47.732829 3968 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.732782 3966 raft_consensus.cc:357] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.732973 3966 raft_consensus.cc:383] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.733074 3966 raft_consensus.cc:738] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.733289 3966 consensus_queue.cc:260] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.733678 3967 tablet_bootstrap.cc:654] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.734608 3968 tablet_bootstrap.cc:654] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.736038 3966 ts_tablet_manager.cc:1428] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250111 02:52:47.736935 3966 tablet_bootstrap.cc:492] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.739136 3966 tablet_bootstrap.cc:654] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.741684 3968 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.741825 3968 ts_tablet_manager.cc:1397] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20250111 02:52:47.742053 3966 tablet_bootstrap.cc:492] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.742177 3966 ts_tablet_manager.cc:1397] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20250111 02:52:47.742424 3967 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.742576 3967 ts_tablet_manager.cc:1397] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20250111 02:52:47.742535 3968 raft_consensus.cc:357] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.742766 3968 raft_consensus.cc:383] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.742858 3968 raft_consensus.cc:738] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.742772 3966 raft_consensus.cc:357] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.743021 3966 raft_consensus.cc:383] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.743126 3966 raft_consensus.cc:738] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.743105 3968 consensus_queue.cc:260] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.743302 3967 raft_consensus.cc:357] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.743363 3966 consensus_queue.cc:260] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.743467 3967 raft_consensus.cc:383] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.743537 3967 raft_consensus.cc:738] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.743623 3968 ts_tablet_manager.cc:1428] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250111 02:52:47.743862 3968 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.743798 3967 consensus_queue.cc:260] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.745512 3968 tablet_bootstrap.cc:654] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.745548 3967 ts_tablet_manager.cc:1428] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250111 02:52:47.745824 3967 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.746342 3966 ts_tablet_manager.cc:1428] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250111 02:52:47.748158 3967 tablet_bootstrap.cc:654] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.748382 3966 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.750016 3966 tablet_bootstrap.cc:654] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.751827 3967 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.751968 3967 ts_tablet_manager.cc:1397] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20250111 02:52:47.752542 3968 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.752712 3968 ts_tablet_manager.cc:1397] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20250111 02:52:47.752710 3967 raft_consensus.cc:357] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.752959 3967 raft_consensus.cc:383] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.753048 3967 raft_consensus.cc:738] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.753249 3967 consensus_queue.cc:260] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.753576 3968 raft_consensus.cc:357] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.753825 3967 ts_tablet_manager.cc:1428] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250111 02:52:47.753866 3968 raft_consensus.cc:383] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.754129 3968 raft_consensus.cc:738] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.754181 3967 tablet_bootstrap.cc:492] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.754484 3968 consensus_queue.cc:260] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.755833 3967 tablet_bootstrap.cc:654] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.756101 3968 ts_tablet_manager.cc:1428] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250111 02:52:47.756440 3968 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.758596 3966 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.758744 3966 ts_tablet_manager.cc:1397] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250111 02:52:47.759425 3975 raft_consensus.cc:491] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.759361 3966 raft_consensus.cc:357] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.759552 3966 raft_consensus.cc:383] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.759660 3968 tablet_bootstrap.cc:654] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.759572 3975 raft_consensus.cc:513] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.759709 3966 raft_consensus.cc:738] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.760017 3966 consensus_queue.cc:260] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.760612 3975 leader_election.cc:290] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:47.762439 3966 ts_tablet_manager.cc:1428] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250111 02:52:47.764743 3966 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.766345 3966 tablet_bootstrap.cc:654] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.769001 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:47.769300 3921 raft_consensus.cc:2463] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:47.769837 3735 leader_election.cc:304] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.770164 3975 raft_consensus.cc:2798] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.770268 3975 raft_consensus.cc:491] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.770377 3975 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.773578 3975 raft_consensus.cc:513] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.774715 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:47.774946 3921 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.776278 3970 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.776961 3970 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.777971 3970 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:47.778892 3921 raft_consensus.cc:2463] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:47.779325 3735 leader_election.cc:304] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.786543 3847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:47.786860 3847 raft_consensus.cc:2463] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:47.787319 3846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:47.787578 3846 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.788365 3967 tablet_bootstrap.cc:492] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.788551 3967 ts_tablet_manager.cc:1397] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.005s
I20250111 02:52:47.789474 3967 raft_consensus.cc:357] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.789721 3967 raft_consensus.cc:383] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.789808 3967 raft_consensus.cc:738] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.790424 3967 consensus_queue.cc:260] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.791111 3846 raft_consensus.cc:2463] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:47.792258 3968 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.792495 3968 ts_tablet_manager.cc:1397] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.030s
I20250111 02:52:47.793493 3968 raft_consensus.cc:357] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.793790 4001 raft_consensus.cc:2798] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:47.793788 3968 raft_consensus.cc:383] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.794010 3968 raft_consensus.cc:738] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.794310 3968 consensus_queue.cc:260] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.794459 3967 ts_tablet_manager.cc:1428] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20250111 02:52:47.795331 3967 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.796612 3968 ts_tablet_manager.cc:1428] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250111 02:52:47.796923 3968 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.797011 3967 tablet_bootstrap.cc:654] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.800442 3975 leader_election.cc:290] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Requested vote from peers 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:47.802024 3968 tablet_bootstrap.cc:654] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.802996 4001 raft_consensus.cc:695] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:47.803330 4001 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:47.806861 3967 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.807024 3967 ts_tablet_manager.cc:1397] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.005s
I20250111 02:52:47.807904 3967 raft_consensus.cc:357] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.808158 3967 raft_consensus.cc:383] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.808255 3967 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.808560 3967 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.809137 3967 ts_tablet_manager.cc:1428] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250111 02:52:47.809423 3967 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.811481 3967 tablet_bootstrap.cc:654] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.814437 3966 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.814599 3966 ts_tablet_manager.cc:1397] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.029s
I20250111 02:52:47.815361 3966 raft_consensus.cc:357] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.815554 3966 raft_consensus.cc:383] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.815639 3966 raft_consensus.cc:738] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.815878 3967 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.816056 3967 ts_tablet_manager.cc:1397] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250111 02:52:47.815954 3668 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } }
I20250111 02:52:47.815936 3966 consensus_queue.cc:260] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.817413 3967 raft_consensus.cc:357] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.817670 3967 raft_consensus.cc:383] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.817766 3967 raft_consensus.cc:738] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.818059 3967 consensus_queue.cc:260] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.819567 3967 ts_tablet_manager.cc:1428] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250111 02:52:47.819855 3967 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.821413 3967 tablet_bootstrap.cc:654] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.821592 3968 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.821785 3968 ts_tablet_manager.cc:1397] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.018s
I20250111 02:52:47.821981 3966 ts_tablet_manager.cc:1428] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250111 02:52:47.822610 3968 raft_consensus.cc:357] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.822947 3968 raft_consensus.cc:383] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.823097 3968 raft_consensus.cc:738] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.823397 3968 consensus_queue.cc:260] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.824337 3966 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.824998 3968 ts_tablet_manager.cc:1428] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250111 02:52:47.825433 3968 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.826166 3966 tablet_bootstrap.cc:654] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.827399 3970 raft_consensus.cc:491] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.827581 3970 raft_consensus.cc:513] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.828282 3970 leader_election.cc:290] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
W20250111 02:52:47.828370 3735 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.103.3:41021, user_credentials={real_user=slave}} blocked reactor thread for 49054.3us
I20250111 02:52:47.829991 3968 tablet_bootstrap.cc:654] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.835300 3966 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.836112 3974 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.836786 3966 ts_tablet_manager.cc:1397] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20250111 02:52:47.837057 3970 raft_consensus.cc:491] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.837210 3970 raft_consensus.cc:513] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.836261 3974 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.838994 3970 leader_election.cc:290] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:47.839982 3966 raft_consensus.cc:357] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.840171 3966 raft_consensus.cc:383] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.840272 3974 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.840256 3966 raft_consensus.cc:738] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.841221 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8" is_pre_election: true
I20250111 02:52:47.847291 3966 consensus_queue.cc:260] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.847512 3772 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 0.
I20250111 02:52:47.841630 3771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8" is_pre_election: true
I20250111 02:52:47.847877 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8" is_pre_election: true
I20250111 02:52:47.848832 3770 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 0.
I20250111 02:52:47.848888 3884 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.849157 3771 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 0.
I20250111 02:52:47.849237 3970 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.849329 3970 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.849442 3970 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.849879 3884 leader_election.cc:304] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.851310 3968 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.851310 3967 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.851537 3968 ts_tablet_manager.cc:1397] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.007s
I20250111 02:52:47.851539 3967 ts_tablet_manager.cc:1397] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.011s
I20250111 02:52:47.852283 3967 raft_consensus.cc:357] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.852461 3967 raft_consensus.cc:383] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.852420 3968 raft_consensus.cc:357] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.852641 3968 raft_consensus.cc:383] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.852756 3968 raft_consensus.cc:738] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.852814 3970 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.852993 3968 consensus_queue.cc:260] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.854406 3968 ts_tablet_manager.cc:1428] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250111 02:52:47.854650 3968 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.855443 3966 ts_tablet_manager.cc:1428] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20250111 02:52:47.855844 3970 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:47.855981 3970 raft_consensus.cc:2798] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.856045 3970 raft_consensus.cc:491] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.856083 3966 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.856109 3970 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.859027 3966 tablet_bootstrap.cc:654] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.862310 3968 tablet_bootstrap.cc:654] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.863904 3884 leader_election.cc:304] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.863947 3967 raft_consensus.cc:738] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.864194 3967 consensus_queue.cc:260] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.864540 3967 ts_tablet_manager.cc:1428] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20250111 02:52:47.864847 3967 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.865417 4012 raft_consensus.cc:2798] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.865499 4012 raft_consensus.cc:491] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.865577 4012 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.865955 3966 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.866062 3967 tablet_bootstrap.cc:654] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.866164 3966 ts_tablet_manager.cc:1397] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20250111 02:52:47.866847 3966 raft_consensus.cc:357] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.867920 3966 raft_consensus.cc:383] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.868032 3966 raft_consensus.cc:738] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.867961 4012 raft_consensus.cc:513] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.868320 3966 consensus_queue.cc:260] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.868798 3771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:47.868960 3771 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.869948 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:47.870098 3770 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.870098 4012 leader_election.cc:290] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:47.874351 3970 raft_consensus.cc:513] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.875699 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:47.875897 3772 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.877009 3770 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 1.
I20250111 02:52:47.877605 3970 leader_election.cc:290] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:47.879427 3884 leader_election.cc:304] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.881150 3970 raft_consensus.cc:2798] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:47.881282 3970 raft_consensus.cc:695] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:47.885362 3771 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 1.
I20250111 02:52:47.887432 3884 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.887974 4012 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:47.888094 4012 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:47.888294 4012 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.889796 3772 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 1.
I20250111 02:52:47.890138 3884 leader_election.cc:304] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.887748 3970 consensus_queue.cc:237] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.894990 3967 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.895107 3967 ts_tablet_manager.cc:1397] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.006s
I20250111 02:52:47.895459 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:47.895711 3921 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 0.
I20250111 02:52:47.895788 3968 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.895890 3968 ts_tablet_manager.cc:1397] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.008s
I20250111 02:52:47.895910 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8" is_pre_election: true
I20250111 02:52:47.896128 3810 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:47.896363 3974 raft_consensus.cc:2798] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.896445 3974 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.896512 3974 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.896692 3968 raft_consensus.cc:357] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.896857 3968 raft_consensus.cc:383] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.896911 3968 raft_consensus.cc:738] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
W20250111 02:52:47.897078 3810 leader_election.cc:343] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:52:47.897086 3968 consensus_queue.cc:260] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.901769 3967 raft_consensus.cc:357] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.901985 3967 raft_consensus.cc:383] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.902056 3967 raft_consensus.cc:738] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.902261 3967 consensus_queue.cc:260] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.904347 3967 ts_tablet_manager.cc:1428] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250111 02:52:47.904644 3967 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.906368 3968 ts_tablet_manager.cc:1428] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250111 02:52:47.906729 3967 tablet_bootstrap.cc:654] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.907402 3968 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: Bootstrap starting.
I20250111 02:52:47.908012 4015 raft_consensus.cc:2798] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:47.909420 3966 ts_tablet_manager.cc:1428] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.043s user 0.011s sys 0.008s
I20250111 02:52:47.910706 3974 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.911616 3846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:47.911882 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:47.911911 3847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:47.912034 3847 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 0.
I20250111 02:52:47.912104 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:47.912240 3843 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
W20250111 02:52:47.912197 3887 leader_election.cc:343] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:52:47.912317 3845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:47.912495 3845 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 0.
I20250111 02:52:47.912791 3966 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: Bootstrap starting.
I20250111 02:52:47.913000 3846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:47.913137 3846 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.913954 3968 tablet_bootstrap.cc:654] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.914316 3966 tablet_bootstrap.cc:654] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.915597 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
W20250111 02:52:47.915937 3887 leader_election.cc:343] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147): Illegal state: must be running to vote when last-logged opid is not known
W20250111 02:52:47.916016 3810 leader_election.cc:343] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:52:47.916435 3974 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.908792 4015 raft_consensus.cc:695] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:47.916791 4015 consensus_queue.cc:237] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.917730 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:47.917869 3921 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.918430 3846 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 1.
I20250111 02:52:47.923692 3843 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 1.
I20250111 02:52:47.919550 3667 catalog_manager.cc:5526] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } }
I20250111 02:52:47.924899 3967 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.927000 3921 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 1.
I20250111 02:52:47.927366 3967 ts_tablet_manager.cc:1397] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.001s
I20250111 02:52:47.928128 3810 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters: 18a7cd9ed0644218857e3da957957cf8
I20250111 02:52:47.928236 3967 raft_consensus.cc:357] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.929112 3967 raft_consensus.cc:383] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.929220 3967 raft_consensus.cc:738] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.929332 3966 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: No bootstrap required, opened a new log
I20250111 02:52:47.929454 3966 ts_tablet_manager.cc:1397] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.007s
I20250111 02:52:47.928524 3974 raft_consensus.cc:2798] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:47.929692 3974 raft_consensus.cc:695] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:47.929921 3974 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.930531 3966 raft_consensus.cc:357] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.930696 3966 raft_consensus.cc:383] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.930756 3966 raft_consensus.cc:738] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.930948 3966 consensus_queue.cc:260] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.930106 3968 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: No bootstrap required, opened a new log
I20250111 02:52:47.931453 3968 ts_tablet_manager.cc:1397] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s
I20250111 02:52:47.929903 3967 consensus_queue.cc:260] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.932251 3968 raft_consensus.cc:357] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.932473 3968 raft_consensus.cc:383] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.932641 3966 ts_tablet_manager.cc:1428] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250111 02:52:47.932649 3968 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.932289 3667 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } }
I20250111 02:52:47.933000 3968 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.932979 3670 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 0 to 1, leader changed from <none> to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 1 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:47.933421 3667 catalog_manager.cc:5526] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } }
I20250111 02:52:47.932493 3967 ts_tablet_manager.cc:1428] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250111 02:52:47.934373 3968 ts_tablet_manager.cc:1428] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:52:47.934417 3967 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8: Bootstrap starting.
I20250111 02:52:47.935654 3967 tablet_bootstrap.cc:654] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8: Neither blocks nor log segments found. Creating new log.
I20250111 02:52:47.938869 3967 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8: No bootstrap required, opened a new log
I20250111 02:52:47.939078 3967 ts_tablet_manager.cc:1397] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250111 02:52:47.939548 3967 raft_consensus.cc:357] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.939672 3967 raft_consensus.cc:383] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:52:47.939733 3967 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Initialized, Role: FOLLOWER
I20250111 02:52:47.939894 3967 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.940312 3967 ts_tablet_manager.cc:1428] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:52:47.966892 3999 raft_consensus.cc:491] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.967057 3999 raft_consensus.cc:513] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.967754 3999 leader_election.cc:290] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.968163 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e886897718de4e46833c86f6412bdf1a" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:47.968186 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e886897718de4e46833c86f6412bdf1a" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:47.968344 3921 raft_consensus.cc:2463] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:47.968361 3843 raft_consensus.cc:2463] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:47.968724 3735 leader_election.cc:304] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.968966 3999 raft_consensus.cc:2798] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.969067 3999 raft_consensus.cc:491] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.969164 3999 raft_consensus.cc:3054] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.970985 3999 raft_consensus.cc:513] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.971467 3999 leader_election.cc:290] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.971819 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e886897718de4e46833c86f6412bdf1a" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:47.971827 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e886897718de4e46833c86f6412bdf1a" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:47.971954 3921 raft_consensus.cc:3054] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.971994 3843 raft_consensus.cc:3054] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.973575 3921 raft_consensus.cc:2463] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:47.973579 3843 raft_consensus.cc:2463] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:47.973932 3735 leader_election.cc:304] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:47.974133 3999 raft_consensus.cc:491] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.974190 4001 raft_consensus.cc:2798] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:47.974315 4001 raft_consensus.cc:695] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:47.974284 3999 raft_consensus.cc:513] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.974563 4001 consensus_queue.cc:237] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.974864 3999 leader_election.cc:290] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.975172 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67b2576c5f45b3b4e291de305ec91c" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:47.975227 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67b2576c5f45b3b4e291de305ec91c" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:47.975353 3843 raft_consensus.cc:2463] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:47.975404 3921 raft_consensus.cc:2463] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:47.975742 3735 leader_election.cc:304] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:47.976038 4001 raft_consensus.cc:2798] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.976140 4001 raft_consensus.cc:491] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.976650 4001 raft_consensus.cc:3054] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.977378 3670 catalog_manager.cc:5526] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:47.978396 4001 raft_consensus.cc:513] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.978940 4001 leader_election.cc:290] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.979272 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67b2576c5f45b3b4e291de305ec91c" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:47.979449 3921 raft_consensus.cc:3054] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.979416 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67b2576c5f45b3b4e291de305ec91c" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:47.979575 3843 raft_consensus.cc:3054] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.981829 3921 raft_consensus.cc:2463] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:47.981829 3843 raft_consensus.cc:2463] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:47.982214 3735 leader_election.cc:304] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:47.982443 4001 raft_consensus.cc:2798] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:47.982568 4001 raft_consensus.cc:695] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:47.982825 4001 consensus_queue.cc:237] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.984659 3670 catalog_manager.cc:5526] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:47.989472 4001 raft_consensus.cc:491] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.989600 4001 raft_consensus.cc:513] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.990097 4001 leader_election.cc:290] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.990247 3974 raft_consensus.cc:491] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:47.990391 3974 raft_consensus.cc:513] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.990469 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358018d07a3d4af0a2102356c90436c0" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:47.990535 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358018d07a3d4af0a2102356c90436c0" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:47.990669 3843 raft_consensus.cc:2463] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:47.990685 3921 raft_consensus.cc:2463] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:47.990929 3974 leader_election.cc:290] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.991068 3738 leader_election.cc:304] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:47.991168 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a4915f163f405fa38c25cc03ceacc9" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8" is_pre_election: true
I20250111 02:52:47.991299 4001 raft_consensus.cc:2798] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.991369 3772 raft_consensus.cc:2463] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 0.
I20250111 02:52:47.991376 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a4915f163f405fa38c25cc03ceacc9" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:47.991679 3920 raft_consensus.cc:2463] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 0.
I20250111 02:52:47.991397 4001 raft_consensus.cc:491] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.991847 3810 leader_election.cc:304] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:47.991858 4001 raft_consensus.cc:3054] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.992102 3974 raft_consensus.cc:2798] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:47.992197 3974 raft_consensus.cc:491] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:47.992291 3974 raft_consensus.cc:3054] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.994534 4001 raft_consensus.cc:513] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.994650 3974 raft_consensus.cc:513] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.995128 3974 leader_election.cc:290] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.995419 4001 leader_election.cc:290] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:47.995500 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a4915f163f405fa38c25cc03ceacc9" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:47.995673 3772 raft_consensus.cc:3054] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.995625 3921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a4915f163f405fa38c25cc03ceacc9" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:47.995819 3921 raft_consensus.cc:3054] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.997354 3772 raft_consensus.cc:2463] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 1.
I20250111 02:52:47.997450 3921 raft_consensus.cc:2463] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 1.
I20250111 02:52:47.997609 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358018d07a3d4af0a2102356c90436c0" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:47.997766 3843 raft_consensus.cc:3054] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.997819 3810 leader_election.cc:304] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:47.998004 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358018d07a3d4af0a2102356c90436c0" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:47.998073 3974 raft_consensus.cc:2798] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:47.998208 3920 raft_consensus.cc:3054] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:47.998600 3974 raft_consensus.cc:695] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:47.998859 3974 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:47.999559 3843 raft_consensus.cc:2463] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:47.999920 3738 leader_election.cc:304] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:48.000456 4001 raft_consensus.cc:2798] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:48.000597 4001 raft_consensus.cc:695] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:48.000857 4001 consensus_queue.cc:237] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.001541 3670 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 0 to 1, leader changed from <none> to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 1 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:48.002238 3920 raft_consensus.cc:2463] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:48.004205 3667 catalog_manager.cc:5526] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:48.036984 4015 raft_consensus.cc:491] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:48.037173 4015 raft_consensus.cc:513] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.037839 4015 leader_election.cc:290] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:48.038249 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be25a13ddfe49af9a73cb6f630f505e" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8" is_pre_election: true
I20250111 02:52:48.038337 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be25a13ddfe49af9a73cb6f630f505e" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:48.038451 3772 raft_consensus.cc:2463] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 0.
I20250111 02:52:48.038544 3843 raft_consensus.cc:2463] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 0.
I20250111 02:52:48.038795 3884 leader_election.cc:304] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:48.039068 4015 raft_consensus.cc:2798] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:48.039180 4015 raft_consensus.cc:491] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:48.039250 4015 raft_consensus.cc:3054] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.040879 4015 raft_consensus.cc:513] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.041410 4015 leader_election.cc:290] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:48.041666 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be25a13ddfe49af9a73cb6f630f505e" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:48.041798 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be25a13ddfe49af9a73cb6f630f505e" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:48.041879 3772 raft_consensus.cc:3054] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.042001 3843 raft_consensus.cc:3054] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.044306 3772 raft_consensus.cc:2463] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 1.
I20250111 02:52:48.044325 3843 raft_consensus.cc:2463] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 1.
I20250111 02:52:48.044653 3884 leader_election.cc:304] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:48.044952 4015 raft_consensus.cc:2798] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:48.045131 4015 raft_consensus.cc:695] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:48.045363 4015 consensus_queue.cc:237] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.047616 3667 catalog_manager.cc:5526] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } }
I20250111 02:52:48.050315 4001 raft_consensus.cc:491] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:48.050454 4001 raft_consensus.cc:513] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.051375 4001 leader_election.cc:290] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:48.051379 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:48.051638 3920 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:48.051735 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:48.051926 3843 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:48.051995 3735 leader_election.cc:304] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:48.052215 4001 raft_consensus.cc:2798] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:48.052312 4001 raft_consensus.cc:491] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:48.052419 4001 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.054015 4001 raft_consensus.cc:513] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.054459 4001 leader_election.cc:290] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:48.054723 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:48.054795 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:48.054906 3843 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.054961 3920 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.056453 3843 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:48.056663 3920 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:48.056819 3738 leader_election.cc:304] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:48.057054 4001 raft_consensus.cc:2798] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:48.057228 4001 raft_consensus.cc:695] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:48.057487 4001 consensus_queue.cc:237] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.059554 3667 catalog_manager.cc:5526] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:48.084452 4001 raft_consensus.cc:491] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:48.084612 4001 raft_consensus.cc:513] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.085197 4001 leader_election.cc:290] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:48.085598 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:48.085619 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:48.085779 3843 raft_consensus.cc:2463] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:48.085819 3920 raft_consensus.cc:2463] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 0.
I20250111 02:52:48.086195 3738 leader_election.cc:304] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:48.086417 4001 raft_consensus.cc:2798] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:52:48.086506 4001 raft_consensus.cc:491] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:52:48.086596 4001 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.088303 4001 raft_consensus.cc:513] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.088820 4001 leader_election.cc:290] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:48.089155 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:48.089192 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:48.089355 3920 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.089358 3843 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.091684 3920 raft_consensus.cc:2463] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:48.091704 3843 raft_consensus.cc:2463] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 1.
I20250111 02:52:48.092129 3738 leader_election.cc:304] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:48.092401 4001 raft_consensus.cc:2798] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:52:48.092525 4001 raft_consensus.cc:695] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:48.092793 4001 consensus_queue.cc:237] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.094928 3667 catalog_manager.cc:5526] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:48.201517 4001 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:52:48.273686 4001 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:48.273972 4001 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.274888 4001 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:48.276085 3846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "912e2307407d45dd88991584cd0b914d" is_pre_election: true
I20250111 02:52:48.278079 3846 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.279892 3920 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.279907 3772 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.281525 3844 raft_consensus.cc:1270] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.282852 3999 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20250111 02:52:48.295132 4001 consensus_queue.cc:1035] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250111 02:52:48.297688 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:48.301718 4020 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20250111 02:52:48.334121 3920 raft_consensus.cc:2388] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18a7cd9ed0644218857e3da957957cf8 in current term 1: Already voted for candidate 912e2307407d45dd88991584cd0b914d in this term.
I20250111 02:52:48.350222 3921 raft_consensus.cc:1270] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.359995 4015 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20250111 02:52:48.364228 3974 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.033s
I20250111 02:52:48.364470 3918 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.362193 4002 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.069s
I20250111 02:52:48.364879 3841 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.365000 3772 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.365059 3845 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.365682 3735 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8; no voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d
I20250111 02:52:48.365754 3847 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.366022 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:48.364951 3919 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.363550 4015 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250111 02:52:48.381013 4055 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:52:48.386199 3974 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250111 02:52:48.389457 3921 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.389585 3970 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250111 02:52:48.389734 3999 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.389861 4001 consensus_queue.cc:1035] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250111 02:52:48.390035 3975 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20250111 02:52:48.403599 3920 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.404079 4002 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20250111 02:52:48.419085 4001 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
W20250111 02:52:48.442456 4033 log.cc:926] Time spent T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
W20250111 02:52:48.446076 3735 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.3:41021, user_credentials={real_user=slave}} blocked reactor thread for 61375.5us
W20250111 02:52:48.446645 4024 log.cc:926] Time spent T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20250111 02:52:48.447106 4012 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20250111 02:52:48.443774 4055 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:48.477006 3770 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.487083 4002 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20250111 02:52:48.487848 4062 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20250111 02:52:48.488524 4018 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250111 02:52:48.496553 3842 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:52:48.496568 3919 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.499938 3847 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.500835 3999 raft_consensus.cc:2743] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:52:48.501111 3771 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.521194 4062 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20250111 02:52:48.497327 4055 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
W20250111 02:52:48.526502 3887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.2:38147, user_credentials={real_user=slave}} blocked reactor thread for 119474us
I20250111 02:52:48.527889 3846 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.531905 3999 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20250111 02:52:48.550621 3771 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.551350 4055 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
W20250111 02:52:48.559360 4071 log.cc:926] Time spent T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20250111 02:52:48.564109 4073 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250111 02:52:48.567224 3970 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20250111 02:52:48.568547 3914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667e917a8b44d0384ced54357111047" is_pre_election: true
I20250111 02:52:48.532351 3767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18a7cd9ed0644218857e3da957957cf8" is_pre_election: true
I20250111 02:52:48.621933 3810 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 912e2307407d45dd88991584cd0b914d; no voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047
I20250111 02:52:48.622005 4046 mvcc.cc:204] Tried to move back new op lower bound from 7112966014494109696 to 7112966012859469824. Current Snapshot: MvccSnapshot[applied={T|T < 7112966014068097024}]
W20250111 02:52:48.679921 3884 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.1:42435, user_credentials={real_user=slave}} blocked reactor thread for 52246.6us
W20250111 02:52:48.686808 4079 log.cc:926] Time spent T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
I20250111 02:52:48.688690 3842 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:52:48.688897 4055 raft_consensus.cc:2743] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250111 02:52:48.710367 4086 log.cc:926] Time spent T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
W20250111 02:52:48.714365 4089 log.cc:926] Time spent T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20250111 02:52:48.723044 4083 log.cc:926] Time spent T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20250111 02:52:48.737087 4065 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20250111 02:52:48.826471 3770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:52:48.835098 3770 raft_consensus.cc:604] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:52:48.864506 3770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:52:48.864810 3770 raft_consensus.cc:604] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:52:48.864846 4076 mvcc.cc:204] Tried to move back new op lower bound from 7112966013886668800 to 7112966012925698048. Current Snapshot: MvccSnapshot[applied={T|T < 7112966013866708992}]
I20250111 02:52:48.865291 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250111 02:52:48.865558 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250111 02:52:48.900010 3750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.907578 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.914743 3747 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.919256 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.933764 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.937604 3747 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.942229 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.943197 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.943764 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.958909 3752 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.963266 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.963438 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.988088 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.988389 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.988732 3750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.989588 3747 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.992413 3750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.994920 3750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.995074 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:48.995743 3747 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.023973 3747 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.024220 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.024968 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.025166 3747 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.025293 3750 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.025974 3745 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.026150 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.025993 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.032378 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.033207 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.034665 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.034729 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.036506 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.037249 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.039040 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.039180 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.039222 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.040212 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.043367 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.043483 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.047173 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.047204 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.047773 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.048599 3748 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.048694 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.048915 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.052197 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.052312 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.054587 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.055274 4001 raft_consensus.cc:988] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:49.055476 4062 raft_consensus.cc:1076] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
W20250111 02:52:49.055702 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.056077 3844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60314
I20250111 02:52:49.056236 3844 raft_consensus.cc:491] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:49.056373 3844 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
W20250111 02:52:49.056798 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.057972 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.058718 3844 raft_consensus.cc:513] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
W20250111 02:52:49.059113 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.059347 3844 leader_election.cc:290] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 2 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:49.059712 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:49.059935 3770 raft_consensus.cc:3049] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:49.059976 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:49.060068 3770 raft_consensus.cc:738] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:49.060140 3916 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
W20250111 02:52:49.060374 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.060336 3770 consensus_queue.cc:260] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:49.060693 3770 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
W20250111 02:52:49.061554 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Illegal state: replica 18a7cd9ed0644218857e3da957957cf8 is not leader of this config: current role FOLLOWER
I20250111 02:52:49.063318 3916 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 2.
W20250111 02:52:49.063455 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Illegal state: replica 18a7cd9ed0644218857e3da957957cf8 is not leader of this config: current role FOLLOWER
I20250111 02:52:49.063516 3770 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 2.
I20250111 02:52:49.063743 3810 leader_election.cc:304] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:49.064020 4060 raft_consensus.cc:2798] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election won for term 2
W20250111 02:52:49.064079 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.064172 4060 raft_consensus.cc:695] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:49.064445 4060 consensus_queue.cc:237] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
W20250111 02:52:49.065120 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Illegal state: replica 18a7cd9ed0644218857e3da957957cf8 is not leader of this config: current role FOLLOWER
I20250111 02:52:49.066648 3670 catalog_manager.cc:5526] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 1 to 2, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 2 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
W20250111 02:52:49.067500 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.068872 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Illegal state: replica 18a7cd9ed0644218857e3da957957cf8 is not leader of this config: current role FOLLOWER
W20250111 02:52:49.069029 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.070318 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.073777 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.075537 3770 raft_consensus.cc:1270] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250111 02:52:49.076574 3916 raft_consensus.cc:1270] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250111 02:52:49.076579 4059 consensus_queue.cc:1035] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250111 02:52:49.077126 4060 consensus_queue.cc:1035] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250111 02:52:49.078954 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.081441 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.083933 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.091427 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.091652 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.093587 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.094657 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.102845 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.106104 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.108177 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.109156 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.118561 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.118654 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.123996 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.125043 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.133371 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.133387 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.137723 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.143092 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.148456 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.148545 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.154731 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.161001 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.167312 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.168362 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.170564 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.178366 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.183529 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.187948 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.188942 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.199301 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.201581 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.206818 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.207819 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.217151 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.223706 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.226876 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.229003 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.237483 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.242789 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.251116 3746 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.251295 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.258494 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
W20250111 02:52:49.262780 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.269309 4001 raft_consensus.cc:988] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:49.269466 4062 raft_consensus.cc:1076] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:49.269976 3844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60314
I20250111 02:52:49.270159 3844 raft_consensus.cc:491] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:49.270259 3844 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
W20250111 02:52:49.272119 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Service unavailable: leader transfer in progress
I20250111 02:52:49.272193 3844 raft_consensus.cc:513] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:49.272745 3844 leader_election.cc:290] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 2 election: Requested vote from peers 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021), 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435)
I20250111 02:52:49.273088 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:49.273273 3916 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:49.273715 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:49.273907 3770 raft_consensus.cc:3049] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:49.273978 3770 raft_consensus.cc:738] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:49.274175 3770 consensus_queue.cc:260] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:49.274394 3770 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:49.275292 3916 raft_consensus.cc:2463] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 2.
W20250111 02:52:49.275575 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Illegal state: replica 18a7cd9ed0644218857e3da957957cf8 is not leader of this config: current role FOLLOWER
I20250111 02:52:49.275743 3810 leader_election.cc:304] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:49.276036 4059 raft_consensus.cc:2798] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:52:49.276188 4059 raft_consensus.cc:695] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:49.276427 4059 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } }
I20250111 02:52:49.276648 3770 raft_consensus.cc:2463] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 2.
I20250111 02:52:49.278445 3668 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 1 to 2, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 2 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } }
W20250111 02:52:49.282920 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Illegal state: replica 18a7cd9ed0644218857e3da957957cf8 is not leader of this config: current role FOLLOWER
W20250111 02:52:49.283855 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Illegal state: replica 18a7cd9ed0644218857e3da957957cf8 is not leader of this config: current role FOLLOWER
W20250111 02:52:49.295106 3743 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44960: Illegal state: replica 18a7cd9ed0644218857e3da957957cf8 is not leader of this config: current role FOLLOWER
W20250111 02:52:49.300385 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58820: Illegal state: replica 4667e917a8b44d0384ced54357111047 is not leader of this config: current role FOLLOWER
W20250111 02:52:49.305553 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58820: Illegal state: replica 4667e917a8b44d0384ced54357111047 is not leader of this config: current role FOLLOWER
W20250111 02:52:49.310007 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58820: Illegal state: replica 4667e917a8b44d0384ced54357111047 is not leader of this config: current role FOLLOWER
W20250111 02:52:49.319196 3901 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58820: Illegal state: replica 4667e917a8b44d0384ced54357111047 is not leader of this config: current role FOLLOWER
I20250111 02:52:49.328095 3916 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250111 02:52:49.328095 3770 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20250111 02:52:49.328617 4060 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250111 02:52:49.328861 4059 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20250111 02:52:49.764212 3810 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.1:42435, user_credentials={real_user=slave}} blocked reactor thread for 57101.4us
W20250111 02:52:49.797441 3887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.2:38147, user_credentials={real_user=slave}} blocked reactor thread for 106422us
I20250111 02:52:49.949108 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:49.953055 4124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:49.960011 4125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:49.961138 4127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:49.961920 412 server_base.cc:1034] running on GCE node
I20250111 02:52:49.962111 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:49.962160 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:49.962188 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563969962185 us; error 0 us; skew 500 ppm
I20250111 02:52:49.962278 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:49.963490 412 webserver.cc:458] Webserver started at http://127.0.103.4:41667/ using document root <none> and password file <none>
I20250111 02:52:49.963702 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:49.963757 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:49.963855 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:49.964555 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-3-root/instance:
uuid: "a9e5558f33db487e97343f676589083e"
format_stamp: "Formatted at 2025-01-11 02:52:49 on dist-test-slave-4xrj"
I20250111 02:52:49.966995 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:52:49.968878 4132 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:49.969204 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:49.969293 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-3-root
uuid: "a9e5558f33db487e97343f676589083e"
format_stamp: "Formatted at 2025-01-11 02:52:49 on dist-test-slave-4xrj"
I20250111 02:52:49.969385 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:49.977996 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:49.978426 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:49.978891 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:49.979594 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:49.979647 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:49.979707 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:49.979744 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:49.996198 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.4:43317
I20250111 02:52:49.996225 4194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.4:43317 every 8 connection(s)
I20250111 02:52:49.998466 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:50.002521 4199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:50.003434 4202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:50.004442 4200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:50.005376 412 server_base.cc:1034] running on GCE node
I20250111 02:52:50.005512 4195 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35787
I20250111 02:52:50.005635 4195 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:50.005652 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:50.005714 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:50.005756 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563970005753 us; error 0 us; skew 500 ppm
I20250111 02:52:50.005918 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:50.006053 4195 heartbeater.cc:510] Master 127.0.103.62:35787 requested a full tablet report, sending...
I20250111 02:52:50.006955 3670 ts_manager.cc:194] Registered new tserver with Master: a9e5558f33db487e97343f676589083e (127.0.103.4:43317)
I20250111 02:52:50.007308 412 webserver.cc:458] Webserver started at http://127.0.103.5:45013/ using document root <none> and password file <none>
I20250111 02:52:50.007575 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:50.007643 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:50.007757 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:50.007789 3670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57670
I20250111 02:52:50.008385 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-4-root/instance:
uuid: "5a90130f2bac4d13875c4094c826566d"
format_stamp: "Formatted at 2025-01-11 02:52:50 on dist-test-slave-4xrj"
I20250111 02:52:50.010792 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:52:50.012212 4207 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:50.012502 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:50.012588 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-4-root
uuid: "5a90130f2bac4d13875c4094c826566d"
format_stamp: "Formatted at 2025-01-11 02:52:50 on dist-test-slave-4xrj"
I20250111 02:52:50.012673 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:50.030010 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:50.030507 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:50.030994 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:50.031739 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:50.031796 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:50.031883 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:50.031925 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:50.047245 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.5:41295
I20250111 02:52:50.047288 4269 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.5:41295 every 8 connection(s)
I20250111 02:52:50.049010 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:52:50.052759 4273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:52:50.053807 4274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:50.054733 412 server_base.cc:1034] running on GCE node
W20250111 02:52:50.055523 4276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:52:50.055881 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:52:50.055934 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:52:50.055959 412 hybrid_clock.cc:648] HybridClock initialized: now 1736563970055956 us; error 0 us; skew 500 ppm
I20250111 02:52:50.056098 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:52:50.056180 4270 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35787
I20250111 02:52:50.056299 4270 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:50.056732 4270 heartbeater.cc:510] Master 127.0.103.62:35787 requested a full tablet report, sending...
I20250111 02:52:50.057763 3670 ts_manager.cc:194] Registered new tserver with Master: 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295)
I20250111 02:52:50.058053 412 webserver.cc:458] Webserver started at http://127.0.103.6:35843/ using document root <none> and password file <none>
I20250111 02:52:50.058264 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:52:50.058332 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:52:50.058442 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:52:50.058668 3670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57678
I20250111 02:52:50.059070 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-5-root/instance:
uuid: "8845a0bcc9d74fa19808e003237576f2"
format_stamp: "Formatted at 2025-01-11 02:52:50 on dist-test-slave-4xrj"
I20250111 02:52:50.061329 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:52:50.062781 4281 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:50.063113 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:52:50.063184 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-5-root
uuid: "8845a0bcc9d74fa19808e003237576f2"
format_stamp: "Formatted at 2025-01-11 02:52:50 on dist-test-slave-4xrj"
I20250111 02:52:50.063253 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:52:50.072784 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:52:50.073196 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:52:50.073611 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:52:50.074411 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:52:50.074462 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:50.074522 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:52:50.074563 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:52:50.091583 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.6:44871
I20250111 02:52:50.091634 4343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.6:44871 every 8 connection(s)
I20250111 02:52:50.098242 4344 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35787
I20250111 02:52:50.098362 4344 heartbeater.cc:463] Registering TS with master...
I20250111 02:52:50.098786 4344 heartbeater.cc:510] Master 127.0.103.62:35787 requested a full tablet report, sending...
I20250111 02:52:50.099491 3670 ts_manager.cc:194] Registered new tserver with Master: 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:52:50.100193 3670 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57684
I20250111 02:52:50.585589 3916 consensus_queue.cc:237] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:50.587711 3768 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.588232 4119 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.588380 3839 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.589071 4119 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.590413 4349 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:50.591192 3768 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:50.592168 3839 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
W20250111 02:52:50.595427 3886 consensus_peers.cc:487] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 -> Peer 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295): Couldn't send request to peer 5a90130f2bac4d13875c4094c826566d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1be25a13ddfe49af9a73cb6f630f505e. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:50.595597 3670 catalog_manager.cc:5526] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index -1 to 28, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:50.601833 3839 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:52:50.603850 3768 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250111 02:52:50.604440 4060 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250111 02:52:50.606892 4060 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:50.607203 3768 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:50.610646 3916 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250111 02:52:50.611066 3668 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index -1 to 26, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:50.612205 4357 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250111 02:52:50.613548 3916 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
W20250111 02:52:50.616640 3811 consensus_peers.cc:487] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d -> Peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317): Couldn't send request to peer a9e5558f33db487e97343f676589083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25a4915f163f405fa38c25cc03ceacc9. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:50.622295 3768 consensus_queue.cc:237] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:52:50.623914 3839 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.624425 4112 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.625169 3916 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.625705 4112 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.627064 4112 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:50.631218 3839 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:50.631428 3920 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:50.636456 3668 catalog_manager.cc:5526] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index -1 to 28, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
W20250111 02:52:50.637717 3737 consensus_peers.cc:487] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 -> Peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871): Couldn't send request to peer 8845a0bcc9d74fa19808e003237576f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358018d07a3d4af0a2102356c90436c0. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:50.641571 3916 consensus_queue.cc:237] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:52:50.643556 3768 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.644062 4119 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.644294 3839 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.644958 4366 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.646371 4119 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:50.646994 3768 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:50.647723 3839 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:50.651525 3667 catalog_manager.cc:5526] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index -1 to 28, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:50.651657 3772 consensus_queue.cc:237] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
W20250111 02:52:50.653373 3886 consensus_peers.cc:487] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 -> Peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871): Couldn't send request to peer 8845a0bcc9d74fa19808e003237576f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d08873d252254c8dbda59ce10b6766cb. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:50.654850 3839 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.655210 4362 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.655352 3916 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.655733 4362 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.657733 4362 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:50.658283 3839 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
W20250111 02:52:50.660836 3736 consensus_peers.cc:487] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 -> Peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317): Couldn't send request to peer a9e5558f33db487e97343f676589083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c67b2576c5f45b3b4e291de305ec91c. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:50.662144 3668 catalog_manager.cc:5526] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index -1 to 28, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:50.662509 3916 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:50.662640 3772 consensus_queue.cc:237] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:50.664790 3839 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.665259 4365 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.668160 4365 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:50.668700 3841 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
W20250111 02:52:50.671506 3737 consensus_peers.cc:487] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 -> Peer 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295): Couldn't send request to peer 5a90130f2bac4d13875c4094c826566d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e886897718de4e46833c86f6412bdf1a. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:50.672549 3916 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:50.672580 3668 catalog_manager.cc:5526] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index -1 to 28, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:50.672947 4362 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:50.674420 3916 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:50.866014 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:50.868193 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:50.868369 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:51.008958 4195 heartbeater.cc:502] Master 127.0.103.62:35787 was elected leader, sending a full tablet report...
I20250111 02:52:51.044327 4368 ts_tablet_manager.cc:927] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: Initiating tablet copy from peer 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:51.045069 4368 tablet_copy_client.cc:323] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:38147
I20250111 02:52:51.051379 3857 tablet_copy_service.cc:140] P 912e2307407d45dd88991584cd0b914d: Received BeginTabletCopySession request for tablet 25a4915f163f405fa38c25cc03ceacc9 from peer a9e5558f33db487e97343f676589083e ({username='slave'} at 127.0.0.1:60352)
I20250111 02:52:51.051545 3857 tablet_copy_service.cc:161] P 912e2307407d45dd88991584cd0b914d: Beginning new tablet copy session on tablet 25a4915f163f405fa38c25cc03ceacc9 from peer a9e5558f33db487e97343f676589083e at {username='slave'} at 127.0.0.1:60352: session id = a9e5558f33db487e97343f676589083e-25a4915f163f405fa38c25cc03ceacc9
I20250111 02:52:51.054100 3857 tablet_copy_source_session.cc:215] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:52:51.055142 4368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a4915f163f405fa38c25cc03ceacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:51.059497 4368 tablet_copy_client.cc:806] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 0 data blocks...
I20250111 02:52:51.059700 4368 tablet_copy_client.cc:670] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 1 WAL segments...
I20250111 02:52:51.059757 4270 heartbeater.cc:502] Master 127.0.103.62:35787 was elected leader, sending a full tablet report...
I20250111 02:52:51.061471 4369 tablet_copy_client.cc:1025] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: tablet copy: Injecting latency into file download: 2000
I20250111 02:52:51.101284 4344 heartbeater.cc:502] Master 127.0.103.62:35787 was elected leader, sending a full tablet report...
I20250111 02:52:51.137454 4371 ts_tablet_manager.cc:927] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: Initiating tablet copy from peer 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:51.138767 4371 tablet_copy_client.cc:323] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:41021
I20250111 02:52:51.145774 4375 ts_tablet_manager.cc:927] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: Initiating tablet copy from peer 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435)
I20250111 02:52:51.149398 4375 tablet_copy_client.cc:323] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.1:42435
I20250111 02:52:51.156600 3931 tablet_copy_service.cc:140] P 4667e917a8b44d0384ced54357111047: Received BeginTabletCopySession request for tablet 1be25a13ddfe49af9a73cb6f630f505e from peer 5a90130f2bac4d13875c4094c826566d ({username='slave'} at 127.0.0.1:58838)
I20250111 02:52:51.156751 3931 tablet_copy_service.cc:161] P 4667e917a8b44d0384ced54357111047: Beginning new tablet copy session on tablet 1be25a13ddfe49af9a73cb6f630f505e from peer 5a90130f2bac4d13875c4094c826566d at {username='slave'} at 127.0.0.1:58838: session id = 5a90130f2bac4d13875c4094c826566d-1be25a13ddfe49af9a73cb6f630f505e
I20250111 02:52:51.159438 3931 tablet_copy_source_session.cc:215] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:52:51.160074 3782 tablet_copy_service.cc:140] P 18a7cd9ed0644218857e3da957957cf8: Received BeginTabletCopySession request for tablet 6c67b2576c5f45b3b4e291de305ec91c from peer a9e5558f33db487e97343f676589083e ({username='slave'} at 127.0.0.1:44986)
I20250111 02:52:51.160250 3782 tablet_copy_service.cc:161] P 18a7cd9ed0644218857e3da957957cf8: Beginning new tablet copy session on tablet 6c67b2576c5f45b3b4e291de305ec91c from peer a9e5558f33db487e97343f676589083e at {username='slave'} at 127.0.0.1:44986: session id = a9e5558f33db487e97343f676589083e-6c67b2576c5f45b3b4e291de305ec91c
I20250111 02:52:51.160635 4371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1be25a13ddfe49af9a73cb6f630f505e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:51.162704 3782 tablet_copy_source_session.cc:215] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:52:51.163692 4375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c67b2576c5f45b3b4e291de305ec91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:51.166136 4371 tablet_copy_client.cc:806] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 0 data blocks...
I20250111 02:52:51.166338 4371 tablet_copy_client.cc:670] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 1 WAL segments...
I20250111 02:52:51.167868 4372 tablet_copy_client.cc:1025] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: tablet copy: Injecting latency into file download: 2000
I20250111 02:52:51.168083 4375 tablet_copy_client.cc:806] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 0 data blocks...
I20250111 02:52:51.168246 4375 tablet_copy_client.cc:670] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 1 WAL segments...
I20250111 02:52:51.169421 4376 tablet_copy_client.cc:1025] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: tablet copy: Injecting latency into file download: 2000
I20250111 02:52:51.220149 4380 ts_tablet_manager.cc:927] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: Initiating tablet copy from peer 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435)
I20250111 02:52:51.220778 4380 tablet_copy_client.cc:323] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.1:42435
I20250111 02:52:51.222599 4382 ts_tablet_manager.cc:927] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: Initiating tablet copy from peer 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435)
I20250111 02:52:51.223176 4382 tablet_copy_client.cc:323] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.1:42435
I20250111 02:52:51.223723 4384 ts_tablet_manager.cc:927] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: Initiating tablet copy from peer 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:51.229950 3782 tablet_copy_service.cc:140] P 18a7cd9ed0644218857e3da957957cf8: Received BeginTabletCopySession request for tablet e886897718de4e46833c86f6412bdf1a from peer 5a90130f2bac4d13875c4094c826566d ({username='slave'} at 127.0.0.1:44990)
I20250111 02:52:51.230118 3782 tablet_copy_service.cc:161] P 18a7cd9ed0644218857e3da957957cf8: Beginning new tablet copy session on tablet e886897718de4e46833c86f6412bdf1a from peer 5a90130f2bac4d13875c4094c826566d at {username='slave'} at 127.0.0.1:44990: session id = 5a90130f2bac4d13875c4094c826566d-e886897718de4e46833c86f6412bdf1a
I20250111 02:52:51.230351 4384 tablet_copy_client.cc:323] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:41021
I20250111 02:52:51.233526 3782 tablet_copy_source_session.cc:215] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:52:51.236382 4380 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e886897718de4e46833c86f6412bdf1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:51.250494 3782 tablet_copy_service.cc:140] P 18a7cd9ed0644218857e3da957957cf8: Received BeginTabletCopySession request for tablet 358018d07a3d4af0a2102356c90436c0 from peer 8845a0bcc9d74fa19808e003237576f2 ({username='slave'} at 127.0.0.1:45002)
I20250111 02:52:51.250614 3782 tablet_copy_service.cc:161] P 18a7cd9ed0644218857e3da957957cf8: Beginning new tablet copy session on tablet 358018d07a3d4af0a2102356c90436c0 from peer 8845a0bcc9d74fa19808e003237576f2 at {username='slave'} at 127.0.0.1:45002: session id = 8845a0bcc9d74fa19808e003237576f2-358018d07a3d4af0a2102356c90436c0
I20250111 02:52:51.251648 4380 tablet_copy_client.cc:806] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 0 data blocks...
I20250111 02:52:51.251852 4380 tablet_copy_client.cc:670] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 1 WAL segments...
I20250111 02:52:51.252861 3931 tablet_copy_service.cc:140] P 4667e917a8b44d0384ced54357111047: Received BeginTabletCopySession request for tablet d08873d252254c8dbda59ce10b6766cb from peer 8845a0bcc9d74fa19808e003237576f2 ({username='slave'} at 127.0.0.1:58844)
I20250111 02:52:51.252979 3931 tablet_copy_service.cc:161] P 4667e917a8b44d0384ced54357111047: Beginning new tablet copy session on tablet d08873d252254c8dbda59ce10b6766cb from peer 8845a0bcc9d74fa19808e003237576f2 at {username='slave'} at 127.0.0.1:58844: session id = 8845a0bcc9d74fa19808e003237576f2-d08873d252254c8dbda59ce10b6766cb
I20250111 02:52:51.252995 3782 tablet_copy_source_session.cc:215] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:52:51.254499 4382 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358018d07a3d4af0a2102356c90436c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:51.255105 3931 tablet_copy_source_session.cc:215] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:52:51.257969 4381 tablet_copy_client.cc:1025] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: tablet copy: Injecting latency into file download: 2000
I20250111 02:52:51.258562 4384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d08873d252254c8dbda59ce10b6766cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:51.258924 4382 tablet_copy_client.cc:806] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 0 data blocks...
I20250111 02:52:51.259127 4382 tablet_copy_client.cc:670] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 1 WAL segments...
I20250111 02:52:51.260429 4383 tablet_copy_client.cc:1025] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Injecting latency into file download: 2000
I20250111 02:52:51.262722 4384 tablet_copy_client.cc:806] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 0 data blocks...
I20250111 02:52:51.262909 4384 tablet_copy_client.cc:670] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 1 WAL segments...
I20250111 02:52:51.264142 4386 tablet_copy_client.cc:1025] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Injecting latency into file download: 2000
I20250111 02:52:52.868664 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:52.871562 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:52:52.871757 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:53.061836 4368 tablet_copy_client.cc:538] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:52:53.065537 4368 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: Bootstrap starting.
I20250111 02:52:53.105376 4368 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:52:53.105726 4368 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: Bootstrap complete.
I20250111 02:52:53.106014 4368 ts_tablet_manager.cc:1397] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: Time spent bootstrapping tablet: real 0.041s user 0.036s sys 0.008s
I20250111 02:52:53.106631 4368 raft_consensus.cc:357] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:52:53.106866 4368 raft_consensus.cc:738] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9e5558f33db487e97343f676589083e, State: Initialized, Role: LEARNER
I20250111 02:52:53.107056 4368 consensus_queue.cc:260] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:52:53.107728 4368 ts_tablet_manager.cc:1428] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250111 02:52:53.108332 3857 tablet_copy_service.cc:342] P 912e2307407d45dd88991584cd0b914d: Request end of tablet copy session a9e5558f33db487e97343f676589083e-25a4915f163f405fa38c25cc03ceacc9 received from {username='slave'} at 127.0.0.1:60352
I20250111 02:52:53.108435 3857 tablet_copy_service.cc:434] P 912e2307407d45dd88991584cd0b914d: ending tablet copy session a9e5558f33db487e97343f676589083e-25a4915f163f405fa38c25cc03ceacc9 on tablet 25a4915f163f405fa38c25cc03ceacc9 with peer a9e5558f33db487e97343f676589083e
I20250111 02:52:53.133016 4170 raft_consensus.cc:1212] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20250111 02:52:53.168203 4371 tablet_copy_client.cc:538] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:52:53.169698 4375 tablet_copy_client.cc:538] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:52:53.171521 4371 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: Bootstrap starting.
I20250111 02:52:53.172401 4375 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: Bootstrap starting.
I20250111 02:52:53.206225 4375 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:52:53.206624 4375 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: Bootstrap complete.
I20250111 02:52:53.206913 4375 ts_tablet_manager.cc:1397] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: Time spent bootstrapping tablet: real 0.035s user 0.034s sys 0.001s
I20250111 02:52:53.207592 4375 raft_consensus.cc:357] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:52:53.207785 4375 raft_consensus.cc:738] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a9e5558f33db487e97343f676589083e, State: Initialized, Role: LEARNER
I20250111 02:52:53.207966 4375 consensus_queue.cc:260] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:52:53.208879 4375 ts_tablet_manager.cc:1428] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:52:53.209317 4371 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:52:53.209457 3782 tablet_copy_service.cc:342] P 18a7cd9ed0644218857e3da957957cf8: Request end of tablet copy session a9e5558f33db487e97343f676589083e-6c67b2576c5f45b3b4e291de305ec91c received from {username='slave'} at 127.0.0.1:44986
I20250111 02:52:53.209574 3782 tablet_copy_service.cc:434] P 18a7cd9ed0644218857e3da957957cf8: ending tablet copy session a9e5558f33db487e97343f676589083e-6c67b2576c5f45b3b4e291de305ec91c on tablet 6c67b2576c5f45b3b4e291de305ec91c with peer a9e5558f33db487e97343f676589083e
I20250111 02:52:53.209684 4371 tablet_bootstrap.cc:492] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: Bootstrap complete.
I20250111 02:52:53.209961 4371 ts_tablet_manager.cc:1397] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: Time spent bootstrapping tablet: real 0.039s user 0.025s sys 0.014s
I20250111 02:52:53.210649 4371 raft_consensus.cc:357] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:53.210857 4371 raft_consensus.cc:738] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a90130f2bac4d13875c4094c826566d, State: Initialized, Role: LEARNER
I20250111 02:52:53.211040 4371 consensus_queue.cc:260] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:53.211798 4371 ts_tablet_manager.cc:1428] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250111 02:52:53.212599 3931 tablet_copy_service.cc:342] P 4667e917a8b44d0384ced54357111047: Request end of tablet copy session 5a90130f2bac4d13875c4094c826566d-1be25a13ddfe49af9a73cb6f630f505e received from {username='slave'} at 127.0.0.1:58838
I20250111 02:52:53.212692 3931 tablet_copy_service.cc:434] P 4667e917a8b44d0384ced54357111047: ending tablet copy session 5a90130f2bac4d13875c4094c826566d-1be25a13ddfe49af9a73cb6f630f505e on tablet 1be25a13ddfe49af9a73cb6f630f505e with peer 5a90130f2bac4d13875c4094c826566d
I20250111 02:52:53.218072 4245 raft_consensus.cc:1212] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250111 02:52:53.258306 4380 tablet_copy_client.cc:538] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:52:53.260707 4382 tablet_copy_client.cc:538] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:52:53.261497 4380 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: Bootstrap starting.
I20250111 02:52:53.263669 4382 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap starting.
I20250111 02:52:53.264468 4384 tablet_copy_client.cc:538] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:52:53.267711 4384 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: Bootstrap starting.
I20250111 02:52:53.305217 4382 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:52:53.305732 4382 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap complete.
I20250111 02:52:53.306030 4382 ts_tablet_manager.cc:1397] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: Time spent bootstrapping tablet: real 0.042s user 0.033s sys 0.009s
I20250111 02:52:53.306830 4382 raft_consensus.cc:357] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:52:53.307062 4382 raft_consensus.cc:738] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Initialized, Role: LEARNER
I20250111 02:52:53.307334 4382 consensus_queue.cc:260] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:52:53.308630 4380 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:52:53.309094 4380 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: Bootstrap complete.
I20250111 02:52:53.309379 4380 ts_tablet_manager.cc:1397] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: Time spent bootstrapping tablet: real 0.048s user 0.046s sys 0.000s
I20250111 02:52:53.310290 4380 raft_consensus.cc:357] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:53.310578 4380 raft_consensus.cc:738] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5a90130f2bac4d13875c4094c826566d, State: Initialized, Role: LEARNER
I20250111 02:52:53.310825 4380 consensus_queue.cc:260] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:53.311544 4380 ts_tablet_manager.cc:1428] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20250111 02:52:53.312059 3782 tablet_copy_service.cc:342] P 18a7cd9ed0644218857e3da957957cf8: Request end of tablet copy session 5a90130f2bac4d13875c4094c826566d-e886897718de4e46833c86f6412bdf1a received from {username='slave'} at 127.0.0.1:44990
I20250111 02:52:53.312167 3782 tablet_copy_service.cc:434] P 18a7cd9ed0644218857e3da957957cf8: ending tablet copy session 5a90130f2bac4d13875c4094c826566d-e886897718de4e46833c86f6412bdf1a on tablet e886897718de4e46833c86f6412bdf1a with peer 5a90130f2bac4d13875c4094c826566d
I20250111 02:52:53.321009 4382 ts_tablet_manager.cc:1428] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: Time spent starting tablet: real 0.015s user 0.005s sys 0.007s
I20250111 02:52:53.322077 3782 tablet_copy_service.cc:342] P 18a7cd9ed0644218857e3da957957cf8: Request end of tablet copy session 8845a0bcc9d74fa19808e003237576f2-358018d07a3d4af0a2102356c90436c0 received from {username='slave'} at 127.0.0.1:45002
I20250111 02:52:53.322192 3782 tablet_copy_service.cc:434] P 18a7cd9ed0644218857e3da957957cf8: ending tablet copy session 8845a0bcc9d74fa19808e003237576f2-358018d07a3d4af0a2102356c90436c0 on tablet 358018d07a3d4af0a2102356c90436c0 with peer 8845a0bcc9d74fa19808e003237576f2
I20250111 02:52:53.328529 4384 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:52:53.328850 4384 tablet_bootstrap.cc:492] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: Bootstrap complete.
I20250111 02:52:53.329089 4384 ts_tablet_manager.cc:1397] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: Time spent bootstrapping tablet: real 0.061s user 0.038s sys 0.008s
I20250111 02:52:53.329658 4384 raft_consensus.cc:357] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:52:53.329908 4384 raft_consensus.cc:738] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Initialized, Role: LEARNER
I20250111 02:52:53.330072 4384 consensus_queue.cc:260] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:52:53.330583 4384 ts_tablet_manager.cc:1428] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250111 02:52:53.331115 3931 tablet_copy_service.cc:342] P 4667e917a8b44d0384ced54357111047: Request end of tablet copy session 8845a0bcc9d74fa19808e003237576f2-d08873d252254c8dbda59ce10b6766cb received from {username='slave'} at 127.0.0.1:58844
I20250111 02:52:53.331221 3931 tablet_copy_service.cc:434] P 4667e917a8b44d0384ced54357111047: ending tablet copy session 8845a0bcc9d74fa19808e003237576f2-d08873d252254c8dbda59ce10b6766cb on tablet d08873d252254c8dbda59ce10b6766cb with peer 8845a0bcc9d74fa19808e003237576f2
I20250111 02:52:53.350510 4319 raft_consensus.cc:1212] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250111 02:52:53.460363 4319 raft_consensus.cc:1212] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250111 02:52:53.474853 4170 raft_consensus.cc:1212] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250111 02:52:53.599339 4406 raft_consensus.cc:1059] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: attempting to promote NON_VOTER 5a90130f2bac4d13875c4094c826566d to VOTER
I20250111 02:52:53.600152 4406 consensus_queue.cc:237] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:53.601581 4245 raft_consensus.cc:1212] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.28] Dedup: 1.28->[]
I20250111 02:52:53.603817 3839 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.604228 3772 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.605214 4245 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 1 LEARNER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.605022 4407 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20250111 02:52:53.606071 4392 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:53.606746 4392 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:53.611397 4392 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:53.611861 4245 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:53.617575 3667 catalog_manager.cc:5526] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:52:53.617903 3839 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:53.629911 3772 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:53.637177 3920 consensus_queue.cc:237] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:53.638780 3839 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:53.638859 4245 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:53.639266 4409 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:53.639532 4407 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:53.641477 4409 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:53.642113 3839 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:53.644649 4245 raft_consensus.cc:2949] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:53.647015 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1be25a13ddfe49af9a73cb6f630f505e with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:52:53.647583 3668 catalog_manager.cc:5526] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index 29 to 30, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:53.651147 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1be25a13ddfe49af9a73cb6f630f505e with delete_type TABLET_DATA_TOMBSTONED (TS 18a7cd9ed0644218857e3da957957cf8 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:44914
I20250111 02:52:53.651850 4417 tablet_replica.cc:331] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:52:53.652129 4417 raft_consensus.cc:2238] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:53.652766 4417 raft_consensus.cc:2267] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:53.653769 4417 ts_tablet_manager.cc:1905] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:52:53.658492 4417 ts_tablet_manager.cc:1918] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250111 02:52:53.658600 4417 log.cc:1198] T 1be25a13ddfe49af9a73cb6f630f505e P 18a7cd9ed0644218857e3da957957cf8: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root/wals/1be25a13ddfe49af9a73cb6f630f505e
I20250111 02:52:53.659209 3654 catalog_manager.cc:4872] TS 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435): tablet 1be25a13ddfe49af9a73cb6f630f505e (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:52:53.706266 4418 raft_consensus.cc:1059] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: attempting to promote NON_VOTER a9e5558f33db487e97343f676589083e to VOTER
I20250111 02:52:53.707043 4418 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:52:53.708906 3920 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250111 02:52:53.708951 4170 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 LEARNER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250111 02:52:53.709391 4418 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250111 02:52:53.709571 3772 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250111 02:52:53.709738 4395 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250111 02:52:53.710505 4395 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250111 02:52:53.713227 4405 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:53.713726 3772 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:53.713968 4170 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:53.716743 3920 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:53.718376 3667 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index 26 to 27, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:52:53.723497 3839 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:52:53.725026 3772 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:53.725546 4170 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:52:53.725556 4418 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:53.725970 4395 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:53.727794 4405 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:53.728950 4170 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:53.731209 3657 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 25a4915f163f405fa38c25cc03ceacc9 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:52:53.728430 3772 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:53.733168 3666 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index 27 to 28, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New cstate: current_term: 1 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:52:53.736544 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet 25a4915f163f405fa38c25cc03ceacc9 with delete_type TABLET_DATA_TOMBSTONED (TS 4667e917a8b44d0384ced54357111047 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:58806
I20250111 02:52:53.737332 4423 tablet_replica.cc:331] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:52:53.737612 4423 raft_consensus.cc:2238] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:53.737840 4423 raft_consensus.cc:2267] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:53.738646 4423 ts_tablet_manager.cc:1905] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:52:53.743760 4423 ts_tablet_manager.cc:1918] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20250111 02:52:53.743857 4423 log.cc:1198] T 25a4915f163f405fa38c25cc03ceacc9 P 4667e917a8b44d0384ced54357111047: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root/wals/25a4915f163f405fa38c25cc03ceacc9
I20250111 02:52:53.744583 3654 catalog_manager.cc:4872] TS 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021): tablet 25a4915f163f405fa38c25cc03ceacc9 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:52:53.851801 4424 raft_consensus.cc:1059] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: attempting to promote NON_VOTER 8845a0bcc9d74fa19808e003237576f2 to VOTER
I20250111 02:52:53.852762 4424 consensus_queue.cc:237] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:53.855191 3920 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.855528 3841 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.855655 4427 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:53.855897 4319 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.856153 4427 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:53.856379 4424 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:53.859295 4427 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.859748 4319 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.861721 3841 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.863386 3920 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.865952 3666 catalog_manager.cc:5526] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.870748 3768 consensus_queue.cc:237] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:53.872349 4319 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:53.872354 3920 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:53.872809 4425 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:53.873021 4427 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:53.874990 4424 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.875485 4319 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.875507 3920 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.878793 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 358018d07a3d4af0a2102356c90436c0 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:52:53.879287 3668 catalog_manager.cc:5526] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 29 to 30, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.882619 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 358018d07a3d4af0a2102356c90436c0 with delete_type TABLET_DATA_TOMBSTONED (TS 912e2307407d45dd88991584cd0b914d not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:60308
I20250111 02:52:53.883982 4431 tablet_replica.cc:331] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:52:53.884173 4431 raft_consensus.cc:2238] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:53.884469 4431 raft_consensus.cc:2267] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:53.885241 4431 ts_tablet_manager.cc:1905] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:52:53.889065 4431 ts_tablet_manager.cc:1918] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250111 02:52:53.889155 4431 log.cc:1198] T 358018d07a3d4af0a2102356c90436c0 P 912e2307407d45dd88991584cd0b914d: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root/wals/358018d07a3d4af0a2102356c90436c0
I20250111 02:52:53.889824 3657 catalog_manager.cc:4872] TS 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147): tablet 358018d07a3d4af0a2102356c90436c0 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:52:53.932314 4392 raft_consensus.cc:1059] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: attempting to promote NON_VOTER 8845a0bcc9d74fa19808e003237576f2 to VOTER
I20250111 02:52:53.933112 4392 consensus_queue.cc:237] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:53.935283 4319 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.935292 3841 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.935460 3768 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:53.935760 4409 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:53.936061 4407 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:53.936545 4407 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:53.943169 4409 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.943601 3772 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.947535 3670 catalog_manager.cc:5526] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.949018 3841 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.951216 4319 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.955111 3920 consensus_queue.cc:237] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:53.956732 4319 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:53.956784 3841 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:53.957126 4407 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:53.957360 4392 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:53.959335 4407 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.960211 4319 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.962440 3841 raft_consensus.cc:2949] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.964599 3670 catalog_manager.cc:5526] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 29 to 30, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New cstate: current_term: 1 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:52:53.965487 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet d08873d252254c8dbda59ce10b6766cb with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:52:53.968334 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet d08873d252254c8dbda59ce10b6766cb with delete_type TABLET_DATA_TOMBSTONED (TS 18a7cd9ed0644218857e3da957957cf8 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:44914
I20250111 02:52:53.968591 4417 tablet_replica.cc:331] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:52:53.968766 4417 raft_consensus.cc:2238] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:53.969019 4417 raft_consensus.cc:2267] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:53.969781 4417 ts_tablet_manager.cc:1905] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:52:53.974367 4417 ts_tablet_manager.cc:1918] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250111 02:52:53.974457 4417 log.cc:1198] T d08873d252254c8dbda59ce10b6766cb P 18a7cd9ed0644218857e3da957957cf8: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root/wals/d08873d252254c8dbda59ce10b6766cb
I20250111 02:52:53.975051 3654 catalog_manager.cc:4872] TS 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435): tablet d08873d252254c8dbda59ce10b6766cb (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:52:54.080761 4390 raft_consensus.cc:1059] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: attempting to promote NON_VOTER a9e5558f33db487e97343f676589083e to VOTER
I20250111 02:52:54.081614 4390 consensus_queue.cc:237] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:52:54.083892 3920 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:54.084008 4170 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 LEARNER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:54.084443 3841 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:54.084494 4424 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:54.084802 4390 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:54.085355 4425 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:54.088824 4390 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:54.089569 3841 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:54.089890 3916 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:54.093209 4170 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:54.096278 4425 raft_consensus.cc:1059] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: attempting to promote NON_VOTER 5a90130f2bac4d13875c4094c826566d to VOTER
I20250111 02:52:54.097069 4425 consensus_queue.cc:237] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:54.096287 3667 catalog_manager.cc:5526] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index 28 to 29, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:54.100627 4245 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 LEARNER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:54.100803 3916 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:54.101153 4424 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:54.101590 4424 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:54.104054 3841 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:52:54.104492 4427 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:54.105604 4427 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.106455 3916 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.109539 3768 consensus_queue.cc:237] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:52:54.111086 4245 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.112355 4170 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:54.113052 4425 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:54.114362 3839 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:54.114586 3666 catalog_manager.cc:5526] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.115026 4424 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:54.117146 4424 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:54.120329 3839 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.121482 3772 consensus_queue.cc:237] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:54.122610 4170 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:54.124315 3841 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:54.127831 4245 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:54.128077 3916 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250111 02:52:54.128274 4390 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:54.128532 4425 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:54.128722 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 6c67b2576c5f45b3b4e291de305ec91c with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:52:54.130957 4425 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.131314 3666 catalog_manager.cc:5526] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e reported cstate change: config changed from index 29 to 30, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:52:54.134637 4245 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.135771 3916 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.138267 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c67b2576c5f45b3b4e291de305ec91c with delete_type TABLET_DATA_TOMBSTONED (TS 4667e917a8b44d0384ced54357111047 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:58806
I20250111 02:52:54.138557 4423 tablet_replica.cc:331] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:52:54.138736 4423 raft_consensus.cc:2238] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:54.139040 4423 raft_consensus.cc:2267] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:54.139899 4423 ts_tablet_manager.cc:1905] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:52:54.140065 3670 catalog_manager.cc:5526] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d reported cstate change: config changed from index 29 to 30, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:54.141048 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet e886897718de4e46833c86f6412bdf1a with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:52:54.144201 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet e886897718de4e46833c86f6412bdf1a with delete_type TABLET_DATA_TOMBSTONED (TS 912e2307407d45dd88991584cd0b914d not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:60308
I20250111 02:52:54.144547 4431 tablet_replica.cc:331] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:52:54.144712 4431 raft_consensus.cc:2238] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:54.144920 4431 raft_consensus.cc:2267] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:54.144969 4423 ts_tablet_manager.cc:1918] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250111 02:52:54.145071 4423 log.cc:1198] T 6c67b2576c5f45b3b4e291de305ec91c P 4667e917a8b44d0384ced54357111047: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root/wals/6c67b2576c5f45b3b4e291de305ec91c
I20250111 02:52:54.145692 3654 catalog_manager.cc:4872] TS 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021): tablet 6c67b2576c5f45b3b4e291de305ec91c (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:52:54.145818 4431 ts_tablet_manager.cc:1905] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:52:54.149353 4431 ts_tablet_manager.cc:1918] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.29
I20250111 02:52:54.149434 4431 log.cc:1198] T e886897718de4e46833c86f6412bdf1a P 912e2307407d45dd88991584cd0b914d: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root/wals/e886897718de4e46833c86f6412bdf1a
I20250111 02:52:54.149964 3657 catalog_manager.cc:4872] TS 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147): tablet e886897718de4e46833c86f6412bdf1a (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:52:54.872093 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:54.886097 3841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "25a4915f163f405fa38c25cc03ceacc9"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "a9e5558f33db487e97343f676589083e"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:52:54.886318 3841 raft_consensus.cc:604] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Received request to transfer leadership to TS a9e5558f33db487e97343f676589083e
I20250111 02:52:54.887269 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "358018d07a3d4af0a2102356c90436c0"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "8845a0bcc9d74fa19808e003237576f2"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:52:54.887419 3772 raft_consensus.cc:604] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Received request to transfer leadership to TS 8845a0bcc9d74fa19808e003237576f2
I20250111 02:52:54.888262 3841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:52:54.888408 3841 raft_consensus.cc:604] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Received request to transfer leadership to TS 4667e917a8b44d0384ced54357111047
I20250111 02:52:54.893312 3916 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:48134
I20250111 02:52:54.893453 3916 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:52:54.894235 3916 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:48134
I20250111 02:52:54.894361 3916 raft_consensus.cc:604] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:52:54.895153 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e886897718de4e46833c86f6412bdf1a"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "5a90130f2bac4d13875c4094c826566d"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:52:54.895268 3772 raft_consensus.cc:604] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Received request to transfer leadership to TS 5a90130f2bac4d13875c4094c826566d
I20250111 02:52:54.895591 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250111 02:52:54.895761 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:54.966229 4448 raft_consensus.cc:988] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 5a90130f2bac4d13875c4094c826566d to start an election
I20250111 02:52:54.966390 4444 raft_consensus.cc:1076] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Signalling peer 5a90130f2bac4d13875c4094c826566d to start an election
I20250111 02:52:54.966981 4245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e886897718de4e46833c86f6412bdf1a"
dest_uuid: "5a90130f2bac4d13875c4094c826566d"
from {username='slave'} at 127.0.0.1:40078
I20250111 02:52:54.967135 4245 raft_consensus.cc:491] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:54.967202 4245 raft_consensus.cc:3054] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:54.968894 4245 raft_consensus.cc:513] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:54.969559 4245 leader_election.cc:290] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [CANDIDATE]: Term 2 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:54.969930 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e886897718de4e46833c86f6412bdf1a" candidate_uuid: "5a90130f2bac4d13875c4094c826566d" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:54.970093 3772 raft_consensus.cc:3049] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:54.970151 3772 raft_consensus.cc:738] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:54.970361 3772 consensus_queue.cc:260] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:54.970698 3772 raft_consensus.cc:3054] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:54.970548 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e886897718de4e46833c86f6412bdf1a" candidate_uuid: "5a90130f2bac4d13875c4094c826566d" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:54.970903 3916 raft_consensus.cc:3054] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:54.972424 3772 raft_consensus.cc:2463] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a90130f2bac4d13875c4094c826566d in term 2.
I20250111 02:52:54.972555 3916 raft_consensus.cc:2463] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a90130f2bac4d13875c4094c826566d in term 2.
I20250111 02:52:54.972798 4208 leader_election.cc:304] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 5a90130f2bac4d13875c4094c826566d; no voters:
I20250111 02:52:54.973352 4449 raft_consensus.cc:2798] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:52:54.973672 4449 raft_consensus.cc:695] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 2 LEADER]: Becoming Leader. State: Replica: 5a90130f2bac4d13875c4094c826566d, State: Running, Role: LEADER
I20250111 02:52:54.973937 4449 consensus_queue.cc:237] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:54.976300 4448 raft_consensus.cc:988] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 8845a0bcc9d74fa19808e003237576f2 to start an election
I20250111 02:52:54.976159 3670 catalog_manager.cc:5526] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d reported cstate change: term changed from 1 to 2, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 5a90130f2bac4d13875c4094c826566d (127.0.103.5). New cstate: current_term: 2 leader_uuid: "5a90130f2bac4d13875c4094c826566d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:52:54.976469 4448 raft_consensus.cc:1076] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Signalling peer 8845a0bcc9d74fa19808e003237576f2 to start an election
I20250111 02:52:54.976984 4319 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "358018d07a3d4af0a2102356c90436c0"
dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
from {username='slave'} at 127.0.0.1:44632
I20250111 02:52:54.977138 4319 raft_consensus.cc:491] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:54.977245 4319 raft_consensus.cc:3054] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:54.979694 4319 raft_consensus.cc:513] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:54.980451 4319 leader_election.cc:290] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [CANDIDATE]: Term 2 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:54.980891 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358018d07a3d4af0a2102356c90436c0" candidate_uuid: "8845a0bcc9d74fa19808e003237576f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:54.981065 3772 raft_consensus.cc:3049] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:54.981139 3772 raft_consensus.cc:738] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:54.981339 3772 consensus_queue.cc:260] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:54.981639 3772 raft_consensus.cc:3054] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:54.982012 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "358018d07a3d4af0a2102356c90436c0" candidate_uuid: "8845a0bcc9d74fa19808e003237576f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:54.982167 3916 raft_consensus.cc:3054] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:54.983923 3772 raft_consensus.cc:2463] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8845a0bcc9d74fa19808e003237576f2 in term 2.
I20250111 02:52:54.984112 3916 raft_consensus.cc:2463] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8845a0bcc9d74fa19808e003237576f2 in term 2.
I20250111 02:52:54.984268 4282 leader_election.cc:304] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 8845a0bcc9d74fa19808e003237576f2; no voters:
I20250111 02:52:54.984860 4455 raft_consensus.cc:2798] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:52:54.985252 4455 raft_consensus.cc:695] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEADER]: Becoming Leader. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Running, Role: LEADER
I20250111 02:52:54.985494 4455 consensus_queue.cc:237] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:54.987711 3670 catalog_manager.cc:5526] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: term changed from 1 to 2, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6). New cstate: current_term: 2 leader_uuid: "8845a0bcc9d74fa19808e003237576f2" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:52:55.053550 4454 raft_consensus.cc:988] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:52:55.053701 4454 raft_consensus.cc:1076] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:52:55.054235 3772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44928
I20250111 02:52:55.054390 3772 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:55.054455 3772 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.056170 3772 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:55.056653 3772 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 2 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:55.057037 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:55.057070 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:55.057174 3841 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.057235 3916 raft_consensus.cc:3049] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:55.057292 3916 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:55.057478 3916 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:55.057732 3916 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.059155 3841 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 2.
I20250111 02:52:55.059495 3916 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 2.
I20250111 02:52:55.059556 3738 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:55.059794 4448 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:52:55.059888 4448 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:55.060091 4448 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:55.062531 3670 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 1 to 2, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 2 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:55.107896 4418 raft_consensus.cc:988] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:52:55.108073 4418 raft_consensus.cc:1076] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Signalling peer 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:52:55.108675 3916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:58816
I20250111 02:52:55.108886 3916 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:55.108987 3916 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.110903 3916 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:55.111435 3916 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 2 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:55.111820 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:55.111927 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:55.111980 3841 raft_consensus.cc:3049] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:55.112093 3841 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:55.112098 3772 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.112372 3841 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:55.112649 3841 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.114027 3772 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 2.
I20250111 02:52:55.114387 3884 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:55.114557 3841 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 2.
I20250111 02:52:55.114614 4454 raft_consensus.cc:2798] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:52:55.114720 4454 raft_consensus.cc:695] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:55.114970 4454 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:55.117230 3670 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 1 to 2, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 2 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } }
I20250111 02:52:55.147022 3841 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:55.149452 3916 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250111 02:52:55.149724 3772 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250111 02:52:55.150002 4460 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:55.150249 4418 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:55.153316 4460 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:55.153734 3920 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:55.153735 3772 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index -1 to 29, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
W20250111 02:52:55.158102 3812 consensus_peers.cc:487] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d -> Peer 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295): Couldn't send request to peer 5a90130f2bac4d13875c4094c826566d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e52445200d144479f643c05ecb3a0f8. This is attempt 1: this message will repeat every 5th retry.
W20250111 02:52:55.159395 3718 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20250111 02:52:55.159521 3670 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index -1 to 29, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New cstate: current_term: 2 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
W20250111 02:52:55.162578 3718 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3bf60fc472f742518bd4328f9e88b436, TS 912e2307407d45dd88991584cd0b914d -> TS 8845a0bcc9d74fa19808e003237576f2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
W20250111 02:52:55.162678 3718 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 3bf60fc472f742518bd4328f9e88b436, TS 912e2307407d45dd88991584cd0b914d -> TS 8845a0bcc9d74fa19808e003237576f2 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
W20250111 02:52:55.165163 3718 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b3dc432e24e7449c879c2c35ad1559d4, TS 4667e917a8b44d0384ced54357111047 -> TS a9e5558f33db487e97343f676589083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
W20250111 02:52:55.165228 3718 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b3dc432e24e7449c879c2c35ad1559d4, TS 4667e917a8b44d0384ced54357111047 -> TS a9e5558f33db487e97343f676589083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
W20250111 02:52:55.172168 3718 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e886897718de4e46833c86f6412bdf1a, TS 4667e917a8b44d0384ced54357111047 -> TS 8845a0bcc9d74fa19808e003237576f2 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
W20250111 02:52:55.172241 3718 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e886897718de4e46833c86f6412bdf1a, TS 4667e917a8b44d0384ced54357111047 -> TS 8845a0bcc9d74fa19808e003237576f2 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
W20250111 02:52:55.174476 3718 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 6c67b2576c5f45b3b4e291de305ec91c, TS 912e2307407d45dd88991584cd0b914d -> TS 5a90130f2bac4d13875c4094c826566d move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
W20250111 02:52:55.174551 3718 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 6c67b2576c5f45b3b4e291de305ec91c, TS 912e2307407d45dd88991584cd0b914d -> TS 5a90130f2bac4d13875c4094c826566d move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
W20250111 02:52:55.176537 3718 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0e921916fa0e4883a3028f4fa013bc38, TS 18a7cd9ed0644218857e3da957957cf8 -> TS a9e5558f33db487e97343f676589083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
W20250111 02:52:55.176606 3718 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0e921916fa0e4883a3028f4fa013bc38, TS 18a7cd9ed0644218857e3da957957cf8 -> TS a9e5558f33db487e97343f676589083e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:55.238000 4458 raft_consensus.cc:988] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:55.238185 4458 raft_consensus.cc:1076] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:55.238848 3841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60316
I20250111 02:52:55.239054 3841 raft_consensus.cc:491] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:55.239154 3841 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.241679 3841 raft_consensus.cc:513] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:55.242470 3841 leader_election.cc:290] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 2 election: Requested vote from peers 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:52:55.242792 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:55.249370 3920 raft_consensus.cc:3049] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:55.249531 3920 raft_consensus.cc:738] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:55.249810 3920 consensus_queue.cc:260] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:55.250247 3920 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.251724 4319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:52:55.251892 4319 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.252755 3920 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 2.
I20250111 02:52:55.253141 3810 leader_election.cc:304] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:55.253397 4460 raft_consensus.cc:2798] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:52:55.253516 4460 raft_consensus.cc:695] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:55.253744 4460 consensus_queue.cc:237] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:52:55.254316 4319 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 2.
I20250111 02:52:55.256476 3668 catalog_manager.cc:5526] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 1 to 2, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 2 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:55.331378 4418 raft_consensus.cc:988] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: : Instructing follower a9e5558f33db487e97343f676589083e to start an election
I20250111 02:52:55.331542 4418 raft_consensus.cc:1076] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Signalling peer a9e5558f33db487e97343f676589083e to start an election
I20250111 02:52:55.332116 4170 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "25a4915f163f405fa38c25cc03ceacc9"
dest_uuid: "a9e5558f33db487e97343f676589083e"
from {username='slave'} at 127.0.0.1:46248
I20250111 02:52:55.332285 4170 raft_consensus.cc:491] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:55.332378 4170 raft_consensus.cc:3054] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.334816 4170 raft_consensus.cc:513] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:52:55.335616 4170 leader_election.cc:290] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [CANDIDATE]: Term 2 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:55.335929 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a4915f163f405fa38c25cc03ceacc9" candidate_uuid: "a9e5558f33db487e97343f676589083e" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:55.336011 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25a4915f163f405fa38c25cc03ceacc9" candidate_uuid: "a9e5558f33db487e97343f676589083e" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:55.336113 3772 raft_consensus.cc:3054] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.336119 3841 raft_consensus.cc:3049] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:55.336208 3841 raft_consensus.cc:738] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:55.336431 3841 consensus_queue.cc:260] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:52:55.336738 3841 raft_consensus.cc:3054] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:55.338579 3772 raft_consensus.cc:2463] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e5558f33db487e97343f676589083e in term 2.
I20250111 02:52:55.338910 4133 leader_election.cc:304] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, a9e5558f33db487e97343f676589083e; no voters:
I20250111 02:52:55.339008 3841 raft_consensus.cc:2463] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e5558f33db487e97343f676589083e in term 2.
I20250111 02:52:55.340929 4471 raft_consensus.cc:2798] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:52:55.341347 4471 raft_consensus.cc:695] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Becoming Leader. State: Replica: a9e5558f33db487e97343f676589083e, State: Running, Role: LEADER
I20250111 02:52:55.342043 4471 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:52:55.344379 3668 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e reported cstate change: term changed from 1 to 2, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to a9e5558f33db487e97343f676589083e (127.0.103.4). New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:52:55.519841 3920 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250111 02:52:55.520385 4448 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250111 02:52:55.521684 3772 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:52:55.522200 4474 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:55.536827 3841 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250111 02:52:55.537312 4476 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250111 02:52:55.545033 3920 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:52:55.545578 4484 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:55.561159 3772 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:52:55.561596 4486 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:55.572213 3920 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:52:55.572820 4486 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:55.594770 3772 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250111 02:52:55.595252 4458 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:55.598798 3839 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250111 02:52:55.599315 4454 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:55.738970 4496 ts_tablet_manager.cc:927] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: Initiating tablet copy from peer 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:55.739621 4496 tablet_copy_client.cc:323] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:38147
I20250111 02:52:55.739956 3844 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250111 02:52:55.740518 4471 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:55.746441 3772 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250111 02:52:55.750265 4502 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:55.750418 3857 tablet_copy_service.cc:140] P 912e2307407d45dd88991584cd0b914d: Received BeginTabletCopySession request for tablet 1e52445200d144479f643c05ecb3a0f8 from peer 5a90130f2bac4d13875c4094c826566d ({username='slave'} at 127.0.0.1:43600)
I20250111 02:52:55.750555 3857 tablet_copy_service.cc:161] P 912e2307407d45dd88991584cd0b914d: Beginning new tablet copy session on tablet 1e52445200d144479f643c05ecb3a0f8 from peer 5a90130f2bac4d13875c4094c826566d at {username='slave'} at 127.0.0.1:43600: session id = 5a90130f2bac4d13875c4094c826566d-1e52445200d144479f643c05ecb3a0f8
I20250111 02:52:55.752869 3857 tablet_copy_source_session.cc:215] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:52:55.754020 4496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e52445200d144479f643c05ecb3a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:52:55.760027 4496 tablet_copy_client.cc:806] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 0 data blocks...
I20250111 02:52:55.760274 4496 tablet_copy_client.cc:670] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 1 WAL segments...
I20250111 02:52:55.762086 4497 tablet_copy_client.cc:1025] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Injecting latency into file download: 2000
I20250111 02:52:55.831048 3920 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:52:55.831284 4319 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:52:55.831758 4418 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:55.832257 4418 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:56.896054 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:56.899360 3841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:52:56.899633 3841 raft_consensus.cc:604] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Received request to transfer leadership to TS 4667e917a8b44d0384ced54357111047
I20250111 02:52:56.900637 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:52:56.900820 3772 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:52:56.901829 3920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:48134
I20250111 02:52:56.902015 3920 raft_consensus.cc:604] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:52:56.902410 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250111 02:52:56.902623 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:57.067423 4494 raft_consensus.cc:988] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:52:57.067585 4494 raft_consensus.cc:1076] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:52:57.068141 3772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44928
I20250111 02:52:57.068356 3772 raft_consensus.cc:491] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:57.068459 3772 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:57.070887 3772 raft_consensus.cc:513] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:57.071486 3772 leader_election.cc:290] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 2 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:57.071871 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:57.071943 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:57.072034 3920 raft_consensus.cc:3049] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:52:57.072098 3920 raft_consensus.cc:738] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:57.072124 3844 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:57.072292 3920 consensus_queue.cc:260] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:57.072602 3920 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:52:57.074273 3844 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 2.
I20250111 02:52:57.074600 3738 leader_election.cc:304] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:57.074683 3920 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 2.
I20250111 02:52:57.074865 4512 raft_consensus.cc:2798] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:52:57.075369 4512 raft_consensus.cc:695] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:57.075608 4512 consensus_queue.cc:237] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:57.078647 3666 catalog_manager.cc:5526] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 1 to 2, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 2 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:57.148870 4520 raft_consensus.cc:988] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:57.149015 4520 raft_consensus.cc:1076] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:57.149621 3844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60314
I20250111 02:52:57.149793 3844 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:57.149907 3844 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:57.152034 3844 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:57.152674 3844 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 3 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:57.152998 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:57.153189 3772 raft_consensus.cc:3049] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:52:57.153266 3772 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:57.153203 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:57.153435 3920 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:57.153558 3772 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:57.153908 3772 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:57.155763 3920 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 3.
I20250111 02:52:57.156118 3772 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 3.
I20250111 02:52:57.156121 3810 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:57.156438 4518 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:52:57.156562 4518 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:57.156776 4518 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:57.159543 3666 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 2 to 3, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 3 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:57.236402 4517 raft_consensus.cc:988] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:52:57.236572 4517 raft_consensus.cc:1076] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Signalling peer 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:52:57.237077 3920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8"
dest_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:58816
I20250111 02:52:57.237258 3920 raft_consensus.cc:491] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:57.237351 3920 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:57.239506 3920 raft_consensus.cc:513] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:57.240330 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:57.240377 3920 leader_election.cc:290] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 3 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435)
I20250111 02:52:57.240499 3844 raft_consensus.cc:3049] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:52:57.240463 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e52445200d144479f643c05ecb3a0f8" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:57.240600 3844 raft_consensus.cc:738] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:57.240650 3772 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:57.240859 3844 consensus_queue.cc:260] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:57.241144 3844 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:57.242339 3772 raft_consensus.cc:2463] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 3.
I20250111 02:52:57.242651 3844 raft_consensus.cc:2463] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 3.
I20250111 02:52:57.242679 3884 leader_election.cc:304] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:57.243018 4494 raft_consensus.cc:2798] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:52:57.243144 4494 raft_consensus.cc:695] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:57.243377 4494 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:57.246332 3666 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 2 to 3, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 3 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:57.594139 3844 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250111 02:52:57.594669 4520 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:57.594794 3916 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250111 02:52:57.595252 4520 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:57.659047 3844 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250111 02:52:57.659523 4519 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:57.664990 3772 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
W20250111 02:52:57.665267 3886 consensus_peers.cc:487] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 -> Peer 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295): Couldn't send request to peer 5a90130f2bac4d13875c4094c826566d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:57.665498 4519 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:57.701269 3916 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250111 02:52:57.701756 4517 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:57.706622 3772 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250111 02:52:57.707176 4518 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:52:57.762408 4496 tablet_copy_client.cc:538] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:52:57.765839 4496 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: Bootstrap starting.
I20250111 02:52:57.803570 4496 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:52:57.803970 4496 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: Bootstrap complete.
I20250111 02:52:57.804226 4496 ts_tablet_manager.cc:1397] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: Time spent bootstrapping tablet: real 0.038s user 0.035s sys 0.005s
I20250111 02:52:57.804855 4496 raft_consensus.cc:357] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:57.805141 4496 raft_consensus.cc:738] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5a90130f2bac4d13875c4094c826566d, State: Initialized, Role: LEARNER
I20250111 02:52:57.805326 4496 consensus_queue.cc:260] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:57.805881 4496 ts_tablet_manager.cc:1428] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20250111 02:52:57.806511 3857 tablet_copy_service.cc:342] P 912e2307407d45dd88991584cd0b914d: Request end of tablet copy session 5a90130f2bac4d13875c4094c826566d-1e52445200d144479f643c05ecb3a0f8 received from {username='slave'} at 127.0.0.1:43600
I20250111 02:52:57.806607 3857 tablet_copy_service.cc:434] P 912e2307407d45dd88991584cd0b914d: ending tablet copy session 5a90130f2bac4d13875c4094c826566d-1e52445200d144479f643c05ecb3a0f8 on tablet 1e52445200d144479f643c05ecb3a0f8 with peer 5a90130f2bac4d13875c4094c826566d
I20250111 02:52:58.162478 4245 raft_consensus.cc:3054] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Advancing to term 3
I20250111 02:52:58.679153 4545 raft_consensus.cc:1059] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: attempting to promote NON_VOTER 5a90130f2bac4d13875c4094c826566d to VOTER
I20250111 02:52:58.680137 4545 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:58.682399 3772 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250111 02:52:58.682468 4245 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 LEARNER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250111 02:52:58.682468 3844 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250111 02:52:58.683043 4547 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:58.683360 4545 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:58.683951 4494 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:52:58.687288 4494 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:58.687767 3772 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:58.692227 4245 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:58.692633 3670 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index 29 to 31, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:58.694712 3844 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:58.711863 3920 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:52:58.713709 4245 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250111 02:52:58.713799 3844 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250111 02:52:58.714146 4547 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:52:58.714372 4550 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:52:58.716289 4544 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:58.718251 4245 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:58.720481 3667 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index 31 to 32, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New cstate: current_term: 3 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:52:58.721212 3844 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:52:58.722230 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1e52445200d144479f643c05ecb3a0f8 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:52:58.726269 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e52445200d144479f643c05ecb3a0f8 with delete_type TABLET_DATA_TOMBSTONED (TS 18a7cd9ed0644218857e3da957957cf8 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:44914
I20250111 02:52:58.726841 4555 tablet_replica.cc:331] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:52:58.727023 4555 raft_consensus.cc:2238] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:52:58.727434 4555 raft_consensus.cc:2267] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:52:58.728130 4555 ts_tablet_manager.cc:1905] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:52:58.731842 4555 ts_tablet_manager.cc:1918] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.31
I20250111 02:52:58.731933 4555 log.cc:1198] T 1e52445200d144479f643c05ecb3a0f8 P 18a7cd9ed0644218857e3da957957cf8: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root/wals/1e52445200d144479f643c05ecb3a0f8
I20250111 02:52:58.732548 3654 catalog_manager.cc:4872] TS 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435): tablet 1e52445200d144479f643c05ecb3a0f8 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:52:58.903088 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:52:58.905968 3844 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:52:58.906163 3844 raft_consensus.cc:604] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Received request to transfer leadership to TS 4667e917a8b44d0384ced54357111047
I20250111 02:52:58.907186 3920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:48134
I20250111 02:52:58.907330 3920 raft_consensus.cc:604] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:52:58.908107 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:52:58.908238 3772 raft_consensus.cc:604] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:52:58.908588 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250111 02:52:58.908757 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:52:58.995534 4544 raft_consensus.cc:988] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:52:58.995699 4544 raft_consensus.cc:1076] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:52:58.996234 3772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44928
I20250111 02:52:58.996439 3772 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:58.996536 3772 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:58.998327 3772 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:58.998869 3772 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 3 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:58.999240 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:58.999372 3844 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:58.999316 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:58.999516 3920 raft_consensus.cc:3049] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:52:58.999577 3920 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:58.999826 3920 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:59.000123 3920 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:59.001271 3844 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 3.
I20250111 02:52:59.001595 3738 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:59.001833 4512 raft_consensus.cc:2798] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:52:59.001906 3920 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 3.
I20250111 02:52:59.002210 4512 raft_consensus.cc:695] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 3 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:59.002446 4512 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:59.004475 3667 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 2 to 3, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 3 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:59.022203 4558 raft_consensus.cc:988] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:52:59.022351 4558 raft_consensus.cc:1076] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Signalling peer 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:52:59.022902 3916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38"
dest_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:58816
I20250111 02:52:59.023052 3916 raft_consensus.cc:491] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:59.023114 3916 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:59.024791 3916 raft_consensus.cc:513] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:59.025307 3916 leader_election.cc:290] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 3 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:52:59.025585 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:59.025774 3772 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:59.025895 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:52:59.026063 3844 raft_consensus.cc:3049] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:52:59.026139 3844 raft_consensus.cc:738] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:59.026330 3844 consensus_queue.cc:260] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:59.026623 3844 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:59.027684 3772 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 3.
I20250111 02:52:59.028059 3884 leader_election.cc:304] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:52:59.028314 4544 raft_consensus.cc:2798] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:52:59.028390 3844 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 3.
I20250111 02:52:59.028461 4544 raft_consensus.cc:695] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:52:59.028685 4544 consensus_queue.cc:237] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:59.030938 3667 catalog_manager.cc:5526] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 2 to 3, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 3 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } }
I20250111 02:52:59.080796 4556 raft_consensus.cc:988] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:59.080947 4556 raft_consensus.cc:1076] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:52:59.081516 3844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60314
I20250111 02:52:59.081658 3844 raft_consensus.cc:491] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:52:59.081759 3844 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:59.083436 3844 raft_consensus.cc:513] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:59.083904 3844 leader_election.cc:290] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 3 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:52:59.084256 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:52:59.084309 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:52:59.084391 3772 raft_consensus.cc:3049] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:52:59.084475 3772 raft_consensus.cc:738] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:52:59.084494 3916 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:59.084681 3772 consensus_queue.cc:260] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:59.084952 3772 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:52:59.086825 3916 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 3.
I20250111 02:52:59.087067 3772 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 3.
I20250111 02:52:59.087162 3810 leader_election.cc:304] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:52:59.087391 4558 raft_consensus.cc:2798] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:52:59.087492 4558 raft_consensus.cc:695] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:52:59.087716 4558 consensus_queue.cc:237] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:52:59.090433 3667 catalog_manager.cc:5526] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 2 to 3, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 3 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:59.478826 3844 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250111 02:52:59.479404 4556 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:59.482795 3916 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250111 02:52:59.483613 4558 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:59.484180 3920 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250111 02:52:59.484854 4556 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:59.488269 3772 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250111 02:52:59.488799 4559 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:59.565455 3772 raft_consensus.cc:1270] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250111 02:52:59.566004 4544 consensus_queue.cc:1035] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:59.569770 3844 raft_consensus.cc:1270] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250111 02:52:59.571475 4571 consensus_queue.cc:1035] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:59.732681 4170 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:52:59.735205 3772 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250111 02:52:59.735872 3844 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250111 02:52:59.736014 4576 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:59.736649 4515 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:59.739920 4581 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.740110 3772 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.742046 3844 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
W20250111 02:52:59.743328 4135 consensus_peers.cc:487] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e -> Peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871): Couldn't send request to peer 8845a0bcc9d74fa19808e003237576f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25a4915f163f405fa38c25cc03ceacc9. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:59.745637 3667 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e reported cstate change: config changed from index 28 to 30, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:59.750541 4319 consensus_queue.cc:237] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:52:59.753118 3772 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250111 02:52:59.754773 4586 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:52:59.755795 3920 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250111 02:52:59.756897 4586 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:52:59.757827 4516 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
W20250111 02:52:59.761384 4283 consensus_peers.cc:487] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 -> Peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317): Couldn't send request to peer a9e5558f33db487e97343f676589083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 358018d07a3d4af0a2102356c90436c0. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:59.762867 3844 consensus_queue.cc:237] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:52:59.763083 3666 catalog_manager.cc:5526] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 30 to 32, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New cstate: current_term: 2 leader_uuid: "8845a0bcc9d74fa19808e003237576f2" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250111 02:52:59.764626 3812 consensus_peers.cc:487] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d -> Peer 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295): Couldn't send request to peer 5a90130f2bac4d13875c4094c826566d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7bedf7b15ad48319668c0000adaf421. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:59.765358 3772 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250111 02:52:59.765477 3768 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:59.765928 3916 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250111 02:52:59.766862 4559 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:59.767459 4559 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:52:59.771333 4558 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:59.772141 3916 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:59.772141 3768 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index -1 to 30, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:52:59.776870 3668 catalog_manager.cc:5526] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index -1 to 30, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New cstate: current_term: 3 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:59.778599 4245 consensus_queue.cc:237] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:52:59.783929 3768 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250111 02:52:59.784615 4509 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:52:59.787081 3920 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250111 02:52:59.787597 4592 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:52:59.787688 3920 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:59.789206 4592 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:59.793695 3666 catalog_manager.cc:5526] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d reported cstate change: config changed from index 30 to 32, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New cstate: current_term: 2 leader_uuid: "5a90130f2bac4d13875c4094c826566d" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:52:59.793738 3772 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:59.798362 3770 consensus_queue.cc:237] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
W20250111 02:52:59.804078 4209 consensus_peers.cc:487] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d -> Peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317): Couldn't send request to peer a9e5558f33db487e97343f676589083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e886897718de4e46833c86f6412bdf1a. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:59.807721 3916 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:52:59.810460 3844 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20250111 02:52:59.810880 3737 consensus_peers.cc:487] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 -> Peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871): Couldn't send request to peer 8845a0bcc9d74fa19808e003237576f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3dc432e24e7449c879c2c35ad1559d4. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:59.810915 4556 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250111 02:52:59.811430 3920 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250111 02:52:59.811856 4556 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250111 02:52:59.815263 3844 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.814785 4512 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.816779 3920 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.820786 3666 catalog_manager.cc:5526] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index -1 to 27, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.823148 3844 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
W20250111 02:52:59.825547 3812 consensus_peers.cc:487] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d -> Peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871): Couldn't send request to peer 8845a0bcc9d74fa19808e003237576f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cf22853b8e742d98610db9dbcfea1c0. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:52:59.825608 3770 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250111 02:52:59.826851 4558 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:59.826953 3920 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250111 02:52:59.827344 4558 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:52:59.829404 4559 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.830055 3770 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.830842 3920 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:52:59.836422 3667 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index -1 to 29, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New cstate: current_term: 3 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:53:00.214979 4606 ts_tablet_manager.cc:927] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: Initiating tablet copy from peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:00.215729 4606 tablet_copy_client.cc:323] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.6:44871
I20250111 02:53:00.216482 4329 tablet_copy_service.cc:140] P 8845a0bcc9d74fa19808e003237576f2: Received BeginTabletCopySession request for tablet 358018d07a3d4af0a2102356c90436c0 from peer a9e5558f33db487e97343f676589083e ({username='slave'} at 127.0.0.1:45052)
I20250111 02:53:00.216684 4329 tablet_copy_service.cc:161] P 8845a0bcc9d74fa19808e003237576f2: Beginning new tablet copy session on tablet 358018d07a3d4af0a2102356c90436c0 from peer a9e5558f33db487e97343f676589083e at {username='slave'} at 127.0.0.1:45052: session id = a9e5558f33db487e97343f676589083e-358018d07a3d4af0a2102356c90436c0
I20250111 02:53:00.218407 4608 ts_tablet_manager.cc:927] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: Initiating tablet copy from peer 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295)
I20250111 02:53:00.219144 4608 tablet_copy_client.cc:323] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.5:41295
I20250111 02:53:00.221015 4329 tablet_copy_source_session.cc:215] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:00.223371 4606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358018d07a3d4af0a2102356c90436c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:00.223546 4611 ts_tablet_manager.cc:927] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: Initiating tablet copy from peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317)
I20250111 02:53:00.230409 4255 tablet_copy_service.cc:140] P 5a90130f2bac4d13875c4094c826566d: Received BeginTabletCopySession request for tablet e886897718de4e46833c86f6412bdf1a from peer a9e5558f33db487e97343f676589083e ({username='slave'} at 127.0.0.1:49680)
I20250111 02:53:00.230598 4255 tablet_copy_service.cc:161] P 5a90130f2bac4d13875c4094c826566d: Beginning new tablet copy session on tablet e886897718de4e46833c86f6412bdf1a from peer a9e5558f33db487e97343f676589083e at {username='slave'} at 127.0.0.1:49680: session id = a9e5558f33db487e97343f676589083e-e886897718de4e46833c86f6412bdf1a
I20250111 02:53:00.232048 4606 tablet_copy_client.cc:806] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:00.232249 4606 tablet_copy_client.cc:670] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:00.233157 4255 tablet_copy_source_session.cc:215] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:00.233817 4607 tablet_copy_client.cc:1025] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:00.234207 4608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e886897718de4e46833c86f6412bdf1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:00.234200 4611 tablet_copy_client.cc:323] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.4:43317
I20250111 02:53:00.234838 4180 tablet_copy_service.cc:140] P a9e5558f33db487e97343f676589083e: Received BeginTabletCopySession request for tablet 25a4915f163f405fa38c25cc03ceacc9 from peer 8845a0bcc9d74fa19808e003237576f2 ({username='slave'} at 127.0.0.1:35306)
I20250111 02:53:00.235025 4180 tablet_copy_service.cc:161] P a9e5558f33db487e97343f676589083e: Beginning new tablet copy session on tablet 25a4915f163f405fa38c25cc03ceacc9 from peer 8845a0bcc9d74fa19808e003237576f2 at {username='slave'} at 127.0.0.1:35306: session id = 8845a0bcc9d74fa19808e003237576f2-25a4915f163f405fa38c25cc03ceacc9
I20250111 02:53:00.238070 4180 tablet_copy_source_session.cc:215] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:00.239214 4611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a4915f163f405fa38c25cc03ceacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:00.240401 4608 tablet_copy_client.cc:806] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:00.240753 4608 tablet_copy_client.cc:670] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:00.242401 4609 tablet_copy_client.cc:1025] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:00.244856 4611 tablet_copy_client.cc:806] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:00.245095 4611 tablet_copy_client.cc:670] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:00.246237 4612 tablet_copy_client.cc:1025] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:00.304941 4613 ts_tablet_manager.cc:927] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: Initiating tablet copy from peer 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:53:00.305482 4613 tablet_copy_client.cc:323] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:38147
I20250111 02:53:00.311269 3857 tablet_copy_service.cc:140] P 912e2307407d45dd88991584cd0b914d: Received BeginTabletCopySession request for tablet 7cf22853b8e742d98610db9dbcfea1c0 from peer 8845a0bcc9d74fa19808e003237576f2 ({username='slave'} at 127.0.0.1:43602)
I20250111 02:53:00.311398 3857 tablet_copy_service.cc:161] P 912e2307407d45dd88991584cd0b914d: Beginning new tablet copy session on tablet 7cf22853b8e742d98610db9dbcfea1c0 from peer 8845a0bcc9d74fa19808e003237576f2 at {username='slave'} at 127.0.0.1:43602: session id = 8845a0bcc9d74fa19808e003237576f2-7cf22853b8e742d98610db9dbcfea1c0
I20250111 02:53:00.313423 3857 tablet_copy_source_session.cc:215] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:00.314476 4613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf22853b8e742d98610db9dbcfea1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:00.320391 4613 tablet_copy_client.cc:806] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:00.320773 4613 tablet_copy_client.cc:670] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:00.321954 4614 tablet_copy_client.cc:1025] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:00.368351 4617 ts_tablet_manager.cc:927] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: Initiating tablet copy from peer 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:53:00.368958 4617 tablet_copy_client.cc:323] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:38147
I20250111 02:53:00.369541 3857 tablet_copy_service.cc:140] P 912e2307407d45dd88991584cd0b914d: Received BeginTabletCopySession request for tablet e7bedf7b15ad48319668c0000adaf421 from peer 5a90130f2bac4d13875c4094c826566d ({username='slave'} at 127.0.0.1:43600)
I20250111 02:53:00.369673 3857 tablet_copy_service.cc:161] P 912e2307407d45dd88991584cd0b914d: Beginning new tablet copy session on tablet e7bedf7b15ad48319668c0000adaf421 from peer 5a90130f2bac4d13875c4094c826566d at {username='slave'} at 127.0.0.1:43600: session id = 5a90130f2bac4d13875c4094c826566d-e7bedf7b15ad48319668c0000adaf421
I20250111 02:53:00.372260 3857 tablet_copy_source_session.cc:215] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:00.373237 4617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bedf7b15ad48319668c0000adaf421. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:00.378044 4617 tablet_copy_client.cc:806] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:00.378293 4617 tablet_copy_client.cc:670] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:00.379609 4618 tablet_copy_client.cc:1025] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:00.411624 4619 ts_tablet_manager.cc:927] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: Initiating tablet copy from peer 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435)
I20250111 02:53:00.412209 4619 tablet_copy_client.cc:323] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.1:42435
I20250111 02:53:00.412716 3782 tablet_copy_service.cc:140] P 18a7cd9ed0644218857e3da957957cf8: Received BeginTabletCopySession request for tablet b3dc432e24e7449c879c2c35ad1559d4 from peer 8845a0bcc9d74fa19808e003237576f2 ({username='slave'} at 127.0.0.1:45002)
I20250111 02:53:00.412829 3782 tablet_copy_service.cc:161] P 18a7cd9ed0644218857e3da957957cf8: Beginning new tablet copy session on tablet b3dc432e24e7449c879c2c35ad1559d4 from peer 8845a0bcc9d74fa19808e003237576f2 at {username='slave'} at 127.0.0.1:45002: session id = 8845a0bcc9d74fa19808e003237576f2-b3dc432e24e7449c879c2c35ad1559d4
I20250111 02:53:00.414777 3782 tablet_copy_source_session.cc:215] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:00.415686 4619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dc432e24e7449c879c2c35ad1559d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:00.420532 4619 tablet_copy_client.cc:806] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:00.420765 4619 tablet_copy_client.cc:670] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:00.421948 4620 tablet_copy_client.cc:1025] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:00.909117 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:00.911819 3920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:48134
I20250111 02:53:00.912019 3920 raft_consensus.cc:604] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:53:00.912878 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:53:00.913049 3772 raft_consensus.cc:604] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 3 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:53:00.913956 3841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:53:00.914093 3841 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Received request to transfer leadership to TS 4667e917a8b44d0384ced54357111047
I20250111 02:53:00.914477 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250111 02:53:00.914672 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:01.150013 4631 raft_consensus.cc:988] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:01.150175 4631 raft_consensus.cc:1076] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:01.150686 3772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44928
I20250111 02:53:01.150879 3772 raft_consensus.cc:491] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:01.150990 3772 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.153328 3772 raft_consensus.cc:513] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:01.153887 3772 leader_election.cc:290] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 4 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:53:01.154286 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:01.154307 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:53:01.154460 3839 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.154465 3920 raft_consensus.cc:3049] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Stepping down as leader of term 3
I20250111 02:53:01.154549 3920 raft_consensus.cc:738] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:01.154757 3920 consensus_queue.cc:260] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:01.155020 3920 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.156208 3839 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 4.
I20250111 02:53:01.156566 3738 leader_election.cc:304] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:53:01.156716 3920 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 4.
I20250111 02:53:01.157007 4630 raft_consensus.cc:2798] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Leader election won for term 4
I20250111 02:53:01.157130 4630 raft_consensus.cc:695] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:01.157349 4630 consensus_queue.cc:237] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:01.159895 3668 catalog_manager.cc:5526] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 3 to 4, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 4 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:01.231119 4629 raft_consensus.cc:988] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:01.231271 4629 raft_consensus.cc:1076] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 3 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:01.231786 3839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60314
I20250111 02:53:01.231972 3839 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:01.232074 3839 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.234643 3839 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:01.235510 3839 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 4 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:53:01.235873 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:53:01.235873 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:01.236089 3772 raft_consensus.cc:3049] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 3 LEADER]: Stepping down as leader of term 3
I20250111 02:53:01.236160 3772 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:01.236367 3772 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:01.236644 3772 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.236089 3920 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.238862 3772 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 4.
I20250111 02:53:01.239274 3810 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:53:01.239513 4627 raft_consensus.cc:2798] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Leader election won for term 4
I20250111 02:53:01.239630 4627 raft_consensus.cc:695] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:01.239866 4627 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:01.240393 3920 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 4.
I20250111 02:53:01.242305 3668 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 3 to 4, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 4 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:01.410967 4628 raft_consensus.cc:988] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:53:01.411154 4628 raft_consensus.cc:1076] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Signalling peer 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:53:01.411700 3920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:58816
I20250111 02:53:01.411890 3920 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:01.411998 3920 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.414496 3920 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:01.415293 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:01.415393 3920 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 4 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:53:01.415364 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:01.415448 3770 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.415503 3839 raft_consensus.cc:3049] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Stepping down as leader of term 3
I20250111 02:53:01.415556 3839 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:01.415783 3839 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:01.416054 3839 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:01.417007 3770 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 4.
I20250111 02:53:01.417371 3884 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:53:01.417630 3839 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 4.
I20250111 02:53:01.417646 4631 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Leader election won for term 4
I20250111 02:53:01.417775 4631 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:01.418231 4631 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:01.420836 3668 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 3 to 4, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 4 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:01.425629 3920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58806
I20250111 02:53:01.425791 3920 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 LEADER]: Received request to transfer leadership
I20250111 02:53:01.432651 3920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58806
I20250111 02:53:01.432811 3920 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 LEADER]: Received request to transfer leadership
W20250111 02:53:01.433123 3654 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 7cf22853b8e742d98610db9dbcfea1c0 already in progress
I20250111 02:53:01.433997 3920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58806
I20250111 02:53:01.434108 3920 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 LEADER]: Received request to transfer leadership
W20250111 02:53:01.434442 3718 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 7cf22853b8e742d98610db9dbcfea1c0 already in progress
W20250111 02:53:01.434497 3718 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 7cf22853b8e742d98610db9dbcfea1c0 already in progress
I20250111 02:53:01.549252 3920 raft_consensus.cc:1270] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250111 02:53:01.549773 4630 consensus_queue.cc:1035] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:01.555210 3839 raft_consensus.cc:1270] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250111 02:53:01.555676 4630 consensus_queue.cc:1035] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:01.700735 3920 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250111 02:53:01.701225 4628 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:01.705746 3770 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250111 02:53:01.706828 4628 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:01.813747 3844 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250111 02:53:01.814275 4631 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:01.816546 4570 raft_consensus.cc:988] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:01.816758 4631 raft_consensus.cc:1076] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:01.817487 3839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60316
I20250111 02:53:01.817651 3839 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:01.817771 3839 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:01.821301 3839 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:01.822295 3844 raft_consensus.cc:1235] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Rejecting Update request from peer 4667e917a8b44d0384ced54357111047 for earlier term 4. Current term is 5. Ops: []
I20250111 02:53:01.822872 3839 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 5 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:53:01.823390 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:53:01.823789 3770 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250111 02:53:01.824117 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:01.824280 3770 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:01.822806 4631 consensus_queue.cc:1046] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: INVALID_TERM, Last received: 4.30, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:01.826064 3920 raft_consensus.cc:3049] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 LEADER]: Stepping down as leader of term 4
I20250111 02:53:01.826153 3920 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:01.826396 3920 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:01.826694 3770 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 5.
I20250111 02:53:01.826974 4654 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:01.827059 3810 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters:
W20250111 02:53:01.827234 3886 consensus_peers.cc:487] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 -> Peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871): Couldn't send request to peer 8845a0bcc9d74fa19808e003237576f2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:01.827302 4605 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Leader election won for term 5
I20250111 02:53:01.827414 4605 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:01.827711 4605 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:01.828091 3920 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:01.830261 3920 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 5.
I20250111 02:53:01.830399 3666 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 4 to 5, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 5 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:02.234185 4606 tablet_copy_client.cc:538] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:02.237550 4606 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: Bootstrap starting.
I20250111 02:53:02.242736 4608 tablet_copy_client.cc:538] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:02.246115 4608 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: Bootstrap starting.
I20250111 02:53:02.246758 4611 tablet_copy_client.cc:538] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:02.249485 4611 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap starting.
I20250111 02:53:02.282214 3772 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20250111 02:53:02.282835 4628 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:02.291294 4611 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:02.291786 4611 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap complete.
I20250111 02:53:02.292143 4611 ts_tablet_manager.cc:1397] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: Time spent bootstrapping tablet: real 0.043s user 0.030s sys 0.010s
I20250111 02:53:02.292910 4611 raft_consensus.cc:357] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:02.293534 4606 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=114 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:02.297189 4606 tablet_bootstrap.cc:492] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: Bootstrap complete.
I20250111 02:53:02.297928 4606 ts_tablet_manager.cc:1397] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: Time spent bootstrapping tablet: real 0.060s user 0.046s sys 0.005s
I20250111 02:53:02.299082 4606 raft_consensus.cc:357] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:02.293241 4611 raft_consensus.cc:738] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Initialized, Role: LEARNER
I20250111 02:53:02.300307 4606 raft_consensus.cc:738] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a9e5558f33db487e97343f676589083e, State: Initialized, Role: LEARNER
I20250111 02:53:02.300536 4611 consensus_queue.cc:260] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:02.300787 4606 consensus_queue.cc:260] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:02.301175 4611 ts_tablet_manager.cc:1428] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20250111 02:53:02.302862 4180 tablet_copy_service.cc:342] P a9e5558f33db487e97343f676589083e: Request end of tablet copy session 8845a0bcc9d74fa19808e003237576f2-25a4915f163f405fa38c25cc03ceacc9 received from {username='slave'} at 127.0.0.1:35306
I20250111 02:53:02.303028 4180 tablet_copy_service.cc:434] P a9e5558f33db487e97343f676589083e: ending tablet copy session 8845a0bcc9d74fa19808e003237576f2-25a4915f163f405fa38c25cc03ceacc9 on tablet 25a4915f163f405fa38c25cc03ceacc9 with peer 8845a0bcc9d74fa19808e003237576f2
I20250111 02:53:02.303936 4606 ts_tablet_manager.cc:1428] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250111 02:53:02.304689 4608 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:02.305127 4608 tablet_bootstrap.cc:492] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: Bootstrap complete.
I20250111 02:53:02.305938 4608 ts_tablet_manager.cc:1397] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: Time spent bootstrapping tablet: real 0.060s user 0.050s sys 0.000s
I20250111 02:53:02.307878 4608 raft_consensus.cc:357] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:02.308240 4608 raft_consensus.cc:738] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a9e5558f33db487e97343f676589083e, State: Initialized, Role: LEARNER
I20250111 02:53:02.309052 4608 consensus_queue.cc:260] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:02.309208 4329 tablet_copy_service.cc:342] P 8845a0bcc9d74fa19808e003237576f2: Request end of tablet copy session a9e5558f33db487e97343f676589083e-358018d07a3d4af0a2102356c90436c0 received from {username='slave'} at 127.0.0.1:45052
I20250111 02:53:02.309413 4329 tablet_copy_service.cc:434] P 8845a0bcc9d74fa19808e003237576f2: ending tablet copy session a9e5558f33db487e97343f676589083e-358018d07a3d4af0a2102356c90436c0 on tablet 358018d07a3d4af0a2102356c90436c0 with peer a9e5558f33db487e97343f676589083e
I20250111 02:53:02.309741 3920 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
W20250111 02:53:02.310070 3812 consensus_peers.cc:487] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d -> Peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871): Couldn't send request to peer 8845a0bcc9d74fa19808e003237576f2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:02.310689 4605 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:02.310863 4608 ts_tablet_manager.cc:1428] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250111 02:53:02.311406 4255 tablet_copy_service.cc:342] P 5a90130f2bac4d13875c4094c826566d: Request end of tablet copy session a9e5558f33db487e97343f676589083e-e886897718de4e46833c86f6412bdf1a received from {username='slave'} at 127.0.0.1:49680
I20250111 02:53:02.311517 4255 tablet_copy_service.cc:434] P 5a90130f2bac4d13875c4094c826566d: ending tablet copy session a9e5558f33db487e97343f676589083e-e886897718de4e46833c86f6412bdf1a on tablet e886897718de4e46833c86f6412bdf1a with peer a9e5558f33db487e97343f676589083e
I20250111 02:53:02.322256 4613 tablet_copy_client.cc:538] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:02.324857 4613 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap starting.
I20250111 02:53:02.355911 4613 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=119 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:02.356266 4613 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap complete.
I20250111 02:53:02.356526 4613 ts_tablet_manager.cc:1397] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: Time spent bootstrapping tablet: real 0.032s user 0.022s sys 0.009s
I20250111 02:53:02.357285 4613 raft_consensus.cc:357] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:02.357470 4613 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Initialized, Role: LEARNER
I20250111 02:53:02.357702 4613 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:02.358345 4613 ts_tablet_manager.cc:1428] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250111 02:53:02.359068 3857 tablet_copy_service.cc:342] P 912e2307407d45dd88991584cd0b914d: Request end of tablet copy session 8845a0bcc9d74fa19808e003237576f2-7cf22853b8e742d98610db9dbcfea1c0 received from {username='slave'} at 127.0.0.1:43602
I20250111 02:53:02.359205 3857 tablet_copy_service.cc:434] P 912e2307407d45dd88991584cd0b914d: ending tablet copy session 8845a0bcc9d74fa19808e003237576f2-7cf22853b8e742d98610db9dbcfea1c0 on tablet 7cf22853b8e742d98610db9dbcfea1c0 with peer 8845a0bcc9d74fa19808e003237576f2
I20250111 02:53:02.367434 4170 raft_consensus.cc:1212] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20250111 02:53:02.379947 4617 tablet_copy_client.cc:538] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:02.382366 4617 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: Bootstrap starting.
I20250111 02:53:02.414280 4617 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:02.414701 4617 tablet_bootstrap.cc:492] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: Bootstrap complete.
I20250111 02:53:02.414976 4617 ts_tablet_manager.cc:1397] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.006s
I20250111 02:53:02.415668 4617 raft_consensus.cc:357] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:02.415930 4617 raft_consensus.cc:738] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5a90130f2bac4d13875c4094c826566d, State: Initialized, Role: LEARNER
I20250111 02:53:02.416152 4617 consensus_queue.cc:260] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:02.416600 4617 ts_tablet_manager.cc:1428] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250111 02:53:02.417213 3857 tablet_copy_service.cc:342] P 912e2307407d45dd88991584cd0b914d: Request end of tablet copy session 5a90130f2bac4d13875c4094c826566d-e7bedf7b15ad48319668c0000adaf421 received from {username='slave'} at 127.0.0.1:43600
I20250111 02:53:02.417341 3857 tablet_copy_service.cc:434] P 912e2307407d45dd88991584cd0b914d: ending tablet copy session 5a90130f2bac4d13875c4094c826566d-e7bedf7b15ad48319668c0000adaf421 on tablet e7bedf7b15ad48319668c0000adaf421 with peer 5a90130f2bac4d13875c4094c826566d
I20250111 02:53:02.422248 4619 tablet_copy_client.cc:538] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:02.425099 4619 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap starting.
I20250111 02:53:02.458453 4619 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=86 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:02.458847 4619 tablet_bootstrap.cc:492] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap complete.
I20250111 02:53:02.459146 4619 ts_tablet_manager.cc:1397] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.009s
I20250111 02:53:02.459769 4619 raft_consensus.cc:357] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:02.460009 4619 raft_consensus.cc:738] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Initialized, Role: LEARNER
I20250111 02:53:02.460243 4619 consensus_queue.cc:260] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:02.460794 4619 ts_tablet_manager.cc:1428] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250111 02:53:02.461498 3782 tablet_copy_service.cc:342] P 18a7cd9ed0644218857e3da957957cf8: Request end of tablet copy session 8845a0bcc9d74fa19808e003237576f2-b3dc432e24e7449c879c2c35ad1559d4 received from {username='slave'} at 127.0.0.1:45002
I20250111 02:53:02.461642 3782 tablet_copy_service.cc:434] P 18a7cd9ed0644218857e3da957957cf8: ending tablet copy session 8845a0bcc9d74fa19808e003237576f2-b3dc432e24e7449c879c2c35ad1559d4 on tablet b3dc432e24e7449c879c2c35ad1559d4 with peer 8845a0bcc9d74fa19808e003237576f2
I20250111 02:53:02.470088 4170 raft_consensus.cc:1212] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20250111 02:53:02.710552 4318 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 3 LEARNER]: Advancing to term 5
I20250111 02:53:02.713066 4318 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 5 LEARNER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250111 02:53:02.713567 4657 consensus_queue.cc:1037] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:02.719911 4245 raft_consensus.cc:1212] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 3 LEARNER]: Deduplicated request from leader. Original: 3.29->[3.30-3.30] Dedup: 3.30->[]
I20250111 02:53:02.728710 4657 raft_consensus.cc:1059] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: attempting to promote NON_VOTER 8845a0bcc9d74fa19808e003237576f2 to VOTER
I20250111 02:53:02.729558 4657 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 30, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:02.731704 3920 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250111 02:53:02.731773 3770 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250111 02:53:02.732066 4318 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 5 LEARNER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250111 02:53:02.732403 4657 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:02.732617 4605 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:02.732895 4669 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:02.735067 4605 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Committing config change with OpId 5.32: config changed from index 29 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:02.735782 3770 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:02.738329 3920 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:02.739248 4319 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:02.740160 3666 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index 29 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:02.747987 3841 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:02.749660 4319 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250111 02:53:02.750064 3770 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250111 02:53:02.750532 4669 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:02.750936 4657 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:02.753393 4605 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:02.754914 4318 raft_consensus.cc:1212] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20250111 02:53:02.756763 3657 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7cf22853b8e742d98610db9dbcfea1c0 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:02.757182 4319 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:02.757812 3770 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:02.758450 3670 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index 32 to 33, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New cstate: current_term: 5 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:02.763844 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf22853b8e742d98610db9dbcfea1c0 with delete_type TABLET_DATA_TOMBSTONED (TS 4667e917a8b44d0384ced54357111047 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:58806
I20250111 02:53:02.764530 4673 tablet_replica.cc:331] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:02.764747 4673 raft_consensus.cc:2238] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:02.764988 4673 raft_consensus.cc:2267] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:02.765717 4673 ts_tablet_manager.cc:1905] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:02.770823 4673 ts_tablet_manager.cc:1918] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.32
I20250111 02:53:02.770964 4673 log.cc:1198] T 7cf22853b8e742d98610db9dbcfea1c0 P 4667e917a8b44d0384ced54357111047: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root/wals/7cf22853b8e742d98610db9dbcfea1c0
I20250111 02:53:02.771582 3654 catalog_manager.cc:4872] TS 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021): tablet 7cf22853b8e742d98610db9dbcfea1c0 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:02.891767 4643 raft_consensus.cc:1059] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: attempting to promote NON_VOTER a9e5558f33db487e97343f676589083e to VOTER
I20250111 02:53:02.892453 4643 consensus_queue.cc:237] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:02.894346 4170 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 LEARNER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:02.894620 3920 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:02.894716 3770 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:02.894824 4674 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:02.895082 4675 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:02.895341 4677 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:02.898900 4675 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:02.899843 3770 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:02.902477 4170 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:02.902493 3920 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:02.903398 3668 catalog_manager.cc:5526] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d reported cstate change: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5a90130f2bac4d13875c4094c826566d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:02.909404 4244 consensus_queue.cc:237] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:02.910984 4170 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:02.911090 3770 raft_consensus.cc:1270] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 5a90130f2bac4d13875c4094c826566d: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:02.911406 4677 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:02.911790 4675 consensus_queue.cc:1035] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:02.913663 4675 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:02.914984 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:02.914671 3770 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:02.917049 3716 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet e886897718de4e46833c86f6412bdf1a with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:02.918301 4169 raft_consensus.cc:2949] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:02.918587 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6c67b2576c5f45b3b4e291de305ec91c"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "a9e5558f33db487e97343f676589083e"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:53:02.918833 3772 raft_consensus.cc:604] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Received request to transfer leadership to TS a9e5558f33db487e97343f676589083e
I20250111 02:53:02.918833 3666 catalog_manager.cc:5526] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d reported cstate change: config changed from index 33 to 34, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New cstate: current_term: 2 leader_uuid: "5a90130f2bac4d13875c4094c826566d" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:02.919979 3839 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:53:02.920178 3839 raft_consensus.cc:604] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Received request to transfer leadership to TS 4667e917a8b44d0384ced54357111047
I20250111 02:53:02.921090 3839 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:53:02.921270 3839 raft_consensus.cc:604] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:53:02.922020 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250111 02:53:02.922345 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:02.922996 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet e886897718de4e46833c86f6412bdf1a with delete_type TABLET_DATA_TOMBSTONED (TS 4667e917a8b44d0384ced54357111047 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:58806
I20250111 02:53:02.923292 4673 tablet_replica.cc:331] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:02.923475 4673 raft_consensus.cc:2238] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:02.923727 4673 raft_consensus.cc:2267] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:02.924830 4673 ts_tablet_manager.cc:1905] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:02.929687 4673 ts_tablet_manager.cc:1918] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20250111 02:53:02.929822 4673 log.cc:1198] T e886897718de4e46833c86f6412bdf1a P 4667e917a8b44d0384ced54357111047: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root/wals/e886897718de4e46833c86f6412bdf1a
I20250111 02:53:02.930415 3654 catalog_manager.cc:4872] TS 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021): tablet e886897718de4e46833c86f6412bdf1a (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:02.967998 4318 raft_consensus.cc:1212] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20250111 02:53:03.008461 4669 raft_consensus.cc:988] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d: : Instructing follower 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:53:03.008646 4669 raft_consensus.cc:1076] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Signalling peer 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:53:03.009158 3920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb"
dest_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:58816
I20250111 02:53:03.009330 3920 raft_consensus.cc:491] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:03.009442 3920 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:53:03.011528 3920 raft_consensus.cc:513] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:03.012221 3920 leader_election.cc:290] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 3 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:03.012586 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:03.012517 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:03.012802 3839 raft_consensus.cc:3049] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:53:03.012799 4318 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:53:03.013068 3839 raft_consensus.cc:738] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:03.013743 3839 consensus_queue.cc:260] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:03.014114 3839 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:53:03.015702 4318 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 3.
I20250111 02:53:03.016084 3886 leader_election.cc:304] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 8845a0bcc9d74fa19808e003237576f2; no voters:
I20250111 02:53:03.016304 4654 raft_consensus.cc:2798] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:53:03.016757 4654 raft_consensus.cc:695] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:03.017056 4654 consensus_queue.cc:237] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:03.018556 3839 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 3.
I20250111 02:53:03.023471 3666 catalog_manager.cc:5526] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 2 to 3, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 3 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:03.071733 4683 raft_consensus.cc:988] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower a9e5558f33db487e97343f676589083e to start an election
I20250111 02:53:03.071914 4683 raft_consensus.cc:1076] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Signalling peer a9e5558f33db487e97343f676589083e to start an election
I20250111 02:53:03.072475 4169 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c67b2576c5f45b3b4e291de305ec91c"
dest_uuid: "a9e5558f33db487e97343f676589083e"
from {username='slave'} at 127.0.0.1:46252
I20250111 02:53:03.072681 4169 raft_consensus.cc:491] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:03.072810 4169 raft_consensus.cc:3054] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:03.075199 4169 raft_consensus.cc:513] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:03.075704 4169 leader_election.cc:290] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [CANDIDATE]: Term 2 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:53:03.076125 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67b2576c5f45b3b4e291de305ec91c" candidate_uuid: "a9e5558f33db487e97343f676589083e" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:03.076268 3770 raft_consensus.cc:3049] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:53:03.076198 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c67b2576c5f45b3b4e291de305ec91c" candidate_uuid: "a9e5558f33db487e97343f676589083e" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:03.076371 3770 raft_consensus.cc:738] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:03.076438 3839 raft_consensus.cc:3054] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:03.076637 3770 consensus_queue.cc:260] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:03.076993 3770 raft_consensus.cc:3054] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:03.078971 3839 raft_consensus.cc:2463] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e5558f33db487e97343f676589083e in term 2.
I20250111 02:53:03.079334 4136 leader_election.cc:304] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912e2307407d45dd88991584cd0b914d, a9e5558f33db487e97343f676589083e; no voters:
I20250111 02:53:03.079368 3770 raft_consensus.cc:2463] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9e5558f33db487e97343f676589083e in term 2.
I20250111 02:53:03.079634 4638 raft_consensus.cc:2798] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:53:03.080049 4638 raft_consensus.cc:695] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Becoming Leader. State: Replica: a9e5558f33db487e97343f676589083e, State: Running, Role: LEADER
I20250111 02:53:03.080322 4638 consensus_queue.cc:237] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:03.082636 3666 catalog_manager.cc:5526] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e reported cstate change: term changed from 1 to 2, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to a9e5558f33db487e97343f676589083e (127.0.103.4). New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:03.095325 4686 raft_consensus.cc:1059] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: attempting to promote NON_VOTER a9e5558f33db487e97343f676589083e to VOTER
I20250111 02:53:03.096043 4686 consensus_queue.cc:237] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:03.098446 3920 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:03.098867 3770 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:03.098927 4686 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:03.099278 4169 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 LEARNER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:03.100033 4689 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:03.100310 4686 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:03.104085 4689 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:03.104684 4169 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:03.108469 3770 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:03.109191 3666 catalog_manager.cc:5526] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 32 to 33, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8845a0bcc9d74fa19808e003237576f2" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:03.115182 4318 consensus_queue.cc:237] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:03.117097 3770 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:03.117129 4169 raft_consensus.cc:1270] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:03.117830 4689 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:03.118150 4687 consensus_queue.cc:1035] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:03.120081 4686 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:03.120774 3770 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:03.120960 4170 raft_consensus.cc:2949] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:03.124362 3716 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 358018d07a3d4af0a2102356c90436c0 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:03.125409 3666 catalog_manager.cc:5526] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index 33 to 34, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New cstate: current_term: 2 leader_uuid: "8845a0bcc9d74fa19808e003237576f2" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:03.128945 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet 358018d07a3d4af0a2102356c90436c0 with delete_type TABLET_DATA_TOMBSTONED (TS 4667e917a8b44d0384ced54357111047 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:58806
I20250111 02:53:03.129251 4673 tablet_replica.cc:331] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:03.129447 4673 raft_consensus.cc:2238] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:03.129628 4673 pending_rounds.cc:62] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Trying to abort 1 pending ops.
I20250111 02:53:03.129819 4673 pending_rounds.cc:69] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Aborting op as it isn't in flight: id { term: 2 index: 33 } timestamp: 7112966074759852032 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "358018d07a3d4af0a2102356c90436c0" old_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } } new_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } } }
I20250111 02:53:03.130133 4673 raft_consensus.cc:2911] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Aborting config change with OpId 2.33: Aborted: Op aborted
I20250111 02:53:03.130270 4673 raft_consensus.cc:2883] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20250111 02:53:03.130375 4673 raft_consensus.cc:2267] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:03.131074 4673 ts_tablet_manager.cc:1905] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:03.135193 4673 ts_tablet_manager.cc:1918] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20250111 02:53:03.135322 4673 log.cc:1198] T 358018d07a3d4af0a2102356c90436c0 P 4667e917a8b44d0384ced54357111047: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root/wals/358018d07a3d4af0a2102356c90436c0
I20250111 02:53:03.135928 3654 catalog_manager.cc:4872] TS 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021): tablet 358018d07a3d4af0a2102356c90436c0 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:03.147080 4684 raft_consensus.cc:1059] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: attempting to promote NON_VOTER 8845a0bcc9d74fa19808e003237576f2 to VOTER
I20250111 02:53:03.147080 4669 raft_consensus.cc:1059] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: attempting to promote NON_VOTER 5a90130f2bac4d13875c4094c826566d to VOTER
W20250111 02:53:03.147274 4669 raft_consensus.cc:1063] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: Unable to promote non-voter 5a90130f2bac4d13875c4094c826566d: Service unavailable: leader transfer in progress
I20250111 02:53:03.147789 4684 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:03.150143 4318 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEARNER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:53:03.150143 3770 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:53:03.150645 4693 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.150982 3839 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:53:03.151001 4684 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.151504 4684 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.154760 4684 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.155738 3841 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.158479 4318 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.159121 3670 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e reported cstate change: config changed from index 30 to 31, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:03.166173 3772 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.167814 4170 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:03.169432 4318 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250111 02:53:03.170058 3772 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250111 02:53:03.170032 4684 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:03.170583 4638 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:03.172151 4638 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.172923 4318 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.173722 3770 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.175345 3715 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 25a4915f163f405fa38c25cc03ceacc9 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:03.176472 3668 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e reported cstate change: config changed from index 31 to 32, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:03.179898 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 25a4915f163f405fa38c25cc03ceacc9 with delete_type TABLET_DATA_TOMBSTONED (TS 912e2307407d45dd88991584cd0b914d not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:60308
I20250111 02:53:03.180835 4698 tablet_replica.cc:331] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:03.181025 4698 raft_consensus.cc:2238] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:03.181275 4698 raft_consensus.cc:2267] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:03.182322 4698 ts_tablet_manager.cc:1905] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:03.186694 4698 ts_tablet_manager.cc:1918] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20250111 02:53:03.186815 4698 log.cc:1198] T 25a4915f163f405fa38c25cc03ceacc9 P 912e2307407d45dd88991584cd0b914d: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root/wals/25a4915f163f405fa38c25cc03ceacc9
I20250111 02:53:03.187472 3657 catalog_manager.cc:4872] TS 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147): tablet 25a4915f163f405fa38c25cc03ceacc9 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:03.229063 4657 raft_consensus.cc:988] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:03.229257 4657 raft_consensus.cc:1076] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:03.229776 3770 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44944
I20250111 02:53:03.230038 3770 raft_consensus.cc:491] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:03.230165 3770 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:03.232546 3770 raft_consensus.cc:513] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:03.233377 3770 leader_election.cc:290] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 4 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:53:03.233785 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:53:03.233958 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:03.234009 3920 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:03.234136 3841 raft_consensus.cc:3049] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Stepping down as leader of term 3
I20250111 02:53:03.234237 3841 raft_consensus.cc:738] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:03.234524 3841 consensus_queue.cc:260] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:03.234978 3841 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:03.236658 3920 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 4.
I20250111 02:53:03.237064 3735 leader_election.cc:304] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 4667e917a8b44d0384ced54357111047; no voters:
I20250111 02:53:03.237346 3841 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 4.
I20250111 02:53:03.237350 4683 raft_consensus.cc:2798] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Leader election won for term 4
I20250111 02:53:03.237509 4683 raft_consensus.cc:695] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:03.237769 4683 consensus_queue.cc:237] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:03.240537 3668 catalog_manager.cc:5526] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 3 to 4, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 4 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:03.246028 3770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:03.246203 3770 raft_consensus.cc:604] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Received request to transfer leadership
I20250111 02:53:03.248963 3770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:03.249151 3770 raft_consensus.cc:604] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Received request to transfer leadership
I20250111 02:53:03.250205 3770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:03.250326 3770 raft_consensus.cc:604] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Received request to transfer leadership
W20250111 02:53:03.250617 3718 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for e7bedf7b15ad48319668c0000adaf421 already in progress
W20250111 02:53:03.250687 3718 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for e7bedf7b15ad48319668c0000adaf421 already in progress
I20250111 02:53:03.469327 3841 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:53:03.469750 4684 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.482933 3772 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:53:03.483621 4693 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.545434 4318 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250111 02:53:03.545984 4654 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:03.550348 3841 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20250111 02:53:03.550935 4654 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:03.590844 4710 raft_consensus.cc:1059] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: attempting to promote NON_VOTER 8845a0bcc9d74fa19808e003237576f2 to VOTER
I20250111 02:53:03.592265 4710 consensus_queue.cc:237] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:03.594574 3841 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:53:03.594607 3920 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:53:03.594795 4318 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 1 LEARNER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250111 02:53:03.595117 4683 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:53:03.595599 4683 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:53:03.596041 4683 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250111 02:53:03.607524 4710 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.611961 3841 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.612397 4318 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.613730 3920 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.616845 3668 catalog_manager.cc:5526] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index 27 to 28, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:03.623490 3772 consensus_queue.cc:237] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:03.624946 4318 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:53:03.625277 3920 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250111 02:53:03.625401 4712 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:53:03.625669 4710 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250111 02:53:03.627490 4712 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.627993 3916 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.628396 4319 raft_consensus.cc:2949] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:03.630374 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet b3dc432e24e7449c879c2c35ad1559d4 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:03.632467 3670 catalog_manager.cc:5526] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index 28 to 29, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New cstate: current_term: 1 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:03.635308 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet b3dc432e24e7449c879c2c35ad1559d4 with delete_type TABLET_DATA_TOMBSTONED (TS 912e2307407d45dd88991584cd0b914d not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:60308
I20250111 02:53:03.635592 4698 tablet_replica.cc:331] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:03.635782 4698 raft_consensus.cc:2238] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:03.636059 4698 raft_consensus.cc:2267] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:03.636758 4698 ts_tablet_manager.cc:1905] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:03.640825 4698 ts_tablet_manager.cc:1918] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20250111 02:53:03.640918 4698 log.cc:1198] T b3dc432e24e7449c879c2c35ad1559d4 P 912e2307407d45dd88991584cd0b914d: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root/wals/b3dc432e24e7449c879c2c35ad1559d4
I20250111 02:53:03.641472 3657 catalog_manager.cc:4872] TS 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147): tablet b3dc432e24e7449c879c2c35ad1559d4 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:03.648138 3916 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250111 02:53:03.648551 4712 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.650432 4683 raft_consensus.cc:988] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:53:03.650552 4683 raft_consensus.cc:1076] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Signalling peer 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:53:03.651376 3920 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421"
dest_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:58812
I20250111 02:53:03.651531 3920 raft_consensus.cc:491] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:03.651623 3920 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:03.651800 3841 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250111 02:53:03.652238 4245 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 3 LEARNER]: Advancing to term 4
I20250111 02:53:03.653021 4683 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.653985 3920 raft_consensus.cc:513] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:03.654448 3916 raft_consensus.cc:1235] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Rejecting Update request from peer 18a7cd9ed0644218857e3da957957cf8 for earlier term 4. Current term is 5. Ops: []
I20250111 02:53:03.654651 4245 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 4 LEARNER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250111 02:53:03.655169 4683 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.655709 4712 consensus_queue.cc:1046] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: INVALID_TERM, Last received: 4.31, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:03.655804 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:03.655929 4712 raft_consensus.cc:3049] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Stepping down as leader of term 4
I20250111 02:53:03.656006 4712 raft_consensus.cc:738] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:03.656217 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bedf7b15ad48319668c0000adaf421" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 5 candidate_status { last_received { term: 4 index: 31 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:03.656369 3841 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:03.656278 4712 consensus_queue.cc:260] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:03.656602 4712 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:03.658102 3841 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 5.
I20250111 02:53:03.658496 3920 leader_election.cc:290] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 5 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:53:03.658625 3887 leader_election.cc:304] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:53:03.658905 4654 raft_consensus.cc:2798] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Leader election won for term 5
I20250111 02:53:03.659022 4654 raft_consensus.cc:695] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 5 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:03.659240 4654 consensus_queue.cc:237] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.31, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:03.661557 3666 catalog_manager.cc:5526] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 4 to 5, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 5 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:03.661943 3770 raft_consensus.cc:2463] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 5.
I20250111 02:53:04.057193 3841 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250111 02:53:04.057713 4681 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:04.062084 4245 raft_consensus.cc:3054] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 4 LEARNER]: Advancing to term 5
I20250111 02:53:04.063319 3770 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250111 02:53:04.063858 4723 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:04.065110 4245 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 5 LEARNER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250111 02:53:04.066031 4681 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:04.067984 4724 raft_consensus.cc:1059] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047: attempting to promote NON_VOTER 5a90130f2bac4d13875c4094c826566d to VOTER
I20250111 02:53:04.068535 4724 consensus_queue.cc:237] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 31, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:04.070461 3770 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250111 02:53:04.070552 4245 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 5 LEARNER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250111 02:53:04.070920 4723 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:04.071089 4654 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:04.071169 3841 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250111 02:53:04.071589 4654 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:04.073323 4723 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 30 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:04.073832 3770 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 30 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:04.074448 3841 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 30 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:04.077638 3667 catalog_manager.cc:5526] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index 30 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:04.077666 4245 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 30 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:04.083220 3920 consensus_queue.cc:237] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 31, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:04.084378 4245 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 5 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250111 02:53:04.084504 3841 raft_consensus.cc:1270] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250111 02:53:04.084790 4654 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:04.085019 4724 consensus_queue.cc:1035] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:04.086701 4724 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 5 LEADER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:04.087299 3841 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:04.088860 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet e7bedf7b15ad48319668c0000adaf421 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:04.090049 3666 catalog_manager.cc:5526] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index 33 to 34, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New cstate: current_term: 5 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:04.087410 4245 raft_consensus.cc:2949] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:04.093253 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bedf7b15ad48319668c0000adaf421 with delete_type TABLET_DATA_TOMBSTONED (TS 18a7cd9ed0644218857e3da957957cf8 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:04.093837 4725 tablet_replica.cc:331] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:04.094066 4725 raft_consensus.cc:2238] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:04.094291 4725 raft_consensus.cc:2267] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:04.094942 4725 ts_tablet_manager.cc:1905] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:04.098732 4725 ts_tablet_manager.cc:1918] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.33
I20250111 02:53:04.098829 4725 log.cc:1198] T e7bedf7b15ad48319668c0000adaf421 P 18a7cd9ed0644218857e3da957957cf8: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root/wals/e7bedf7b15ad48319668c0000adaf421
I20250111 02:53:04.099399 3654 catalog_manager.cc:4872] TS 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435): tablet e7bedf7b15ad48319668c0000adaf421 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:04.636458 4170 consensus_queue.cc:237] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:04.638845 3841 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
W20250111 02:53:04.638994 4135 consensus_peers.cc:487] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e -> Peer 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295): Couldn't send request to peer 5a90130f2bac4d13875c4094c826566d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6c67b2576c5f45b3b4e291de305ec91c. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:04.638893 3770 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250111 02:53:04.639609 4721 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:04.639921 4727 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:04.642580 3770 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:53:04.642599 3841 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:53:04.642127 4728 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 30 to 32, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:53:04.646555 3666 catalog_manager.cc:5526] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index 30 to 32, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:53:04.649309 3841 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
W20250111 02:53:04.652343 3811 consensus_peers.cc:487] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d -> Peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317): Couldn't send request to peer a9e5558f33db487e97343f676589083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cf22853b8e742d98610db9dbcfea1c0. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:04.652338 3770 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250111 02:53:04.652369 4319 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250111 02:53:04.652874 4737 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:04.653193 4736 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:04.655335 4657 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:53:04.655985 4319 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:53:04.656185 3770 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:53:04.660086 3667 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index 33 to 34, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New cstate: current_term: 5 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:04.660825 3920 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:04.663329 4245 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250111 02:53:04.663983 3841 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20250111 02:53:04.664089 4681 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:04.664445 4724 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:04.667853 4654 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:53:04.668385 3841 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:53:04.673460 4245 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } } }
I20250111 02:53:04.673556 3666 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index 32 to 33, NON_VOTER a9e5558f33db487e97343f676589083e (127.0.103.4) added. New cstate: current_term: 3 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:04.676173 3841 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:04.679203 3920 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250111 02:53:04.679203 3770 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 4 index: 31. (index mismatch)
I20250111 02:53:04.679800 4657 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:04.680088 4736 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20250111 02:53:04.680521 3812 consensus_peers.cc:487] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d -> Peer 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871): Couldn't send request to peer 8845a0bcc9d74fa19808e003237576f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3bf60fc472f742518bd4328f9e88b436. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:04.682634 4737 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 LEADER]: Committing config change with OpId 4.31: config changed from index -1 to 31, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
W20250111 02:53:04.684623 3885 consensus_peers.cc:487] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 -> Peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317): Couldn't send request to peer a9e5558f33db487e97343f676589083e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e52445200d144479f643c05ecb3a0f8. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:04.685998 3770 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index -1 to 31, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:53:04.686899 3670 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index -1 to 31, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New cstate: current_term: 4 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:04.687330 4170 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:04.688653 3920 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Committing config change with OpId 4.31: config changed from index -1 to 31, NON_VOTER 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } } }
I20250111 02:53:04.693665 3770 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:04.693876 4319 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
W20250111 02:53:04.694082 4135 consensus_peers.cc:487] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e -> Peer 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295): Couldn't send request to peer 5a90130f2bac4d13875c4094c826566d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 25a4915f163f405fa38c25cc03ceacc9. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:04.694103 4731 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:04.694392 4727 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:04.696414 4731 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:53:04.696931 4319 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:53:04.697378 3770 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:53:04.700832 3670 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 32 to 33, NON_VOTER 5a90130f2bac4d13875c4094c826566d (127.0.103.5) added. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } } }
I20250111 02:53:04.922703 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:04.925570 3920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1be25a13ddfe49af9a73cb6f630f505e"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:48134
I20250111 02:53:04.925789 3920 raft_consensus.cc:604] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:53:04.926759 3920 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
new_leader_uuid: "8845a0bcc9d74fa19808e003237576f2"
from {username='slave'} at 127.0.0.1:48134
I20250111 02:53:04.926942 3920 raft_consensus.cc:604] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Received request to transfer leadership to TS 8845a0bcc9d74fa19808e003237576f2
I20250111 02:53:04.927273 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250111 02:53:04.927493 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:05.039645 4751 ts_tablet_manager.cc:927] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: Initiating tablet copy from peer 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:53:05.040319 4751 tablet_copy_client.cc:323] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:41021
I20250111 02:53:05.046525 3931 tablet_copy_service.cc:140] P 4667e917a8b44d0384ced54357111047: Received BeginTabletCopySession request for tablet 1e52445200d144479f643c05ecb3a0f8 from peer a9e5558f33db487e97343f676589083e ({username='slave'} at 127.0.0.1:36490)
I20250111 02:53:05.046701 3931 tablet_copy_service.cc:161] P 4667e917a8b44d0384ced54357111047: Beginning new tablet copy session on tablet 1e52445200d144479f643c05ecb3a0f8 from peer a9e5558f33db487e97343f676589083e at {username='slave'} at 127.0.0.1:36490: session id = a9e5558f33db487e97343f676589083e-1e52445200d144479f643c05ecb3a0f8
I20250111 02:53:05.049093 3931 tablet_copy_source_session.cc:215] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:05.050184 4751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e52445200d144479f643c05ecb3a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:05.055312 4751 tablet_copy_client.cc:806] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:05.055521 4751 tablet_copy_client.cc:670] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:05.056775 4752 tablet_copy_client.cc:1025] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:05.058207 4681 raft_consensus.cc:988] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:05.058359 4681 raft_consensus.cc:1076] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:05.058833 3839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1be25a13ddfe49af9a73cb6f630f505e"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60316
I20250111 02:53:05.059031 3839 raft_consensus.cc:491] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:05.059173 3839 raft_consensus.cc:3054] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:05.061268 3839 raft_consensus.cc:513] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:05.062096 3839 leader_election.cc:290] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 2 election: Requested vote from peers 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021), 5a90130f2bac4d13875c4094c826566d (127.0.103.5:41295)
I20250111 02:53:05.062505 4245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be25a13ddfe49af9a73cb6f630f505e" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "5a90130f2bac4d13875c4094c826566d"
I20250111 02:53:05.062769 4245 raft_consensus.cc:3054] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:05.064841 4245 raft_consensus.cc:2463] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 2.
I20250111 02:53:05.062565 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1be25a13ddfe49af9a73cb6f630f505e" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:53:05.065287 3812 leader_election.cc:304] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a90130f2bac4d13875c4094c826566d, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:53:05.065287 3920 raft_consensus.cc:3049] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:53:05.065479 3920 raft_consensus.cc:738] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:05.065616 4657 raft_consensus.cc:2798] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:53:05.065752 4657 raft_consensus.cc:695] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:05.066015 4657 consensus_queue.cc:237] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:05.065762 3920 consensus_queue.cc:260] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:05.067281 3920 raft_consensus.cc:3054] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:05.075480 3666 catalog_manager.cc:5526] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 1 to 2, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 2 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:05.086407 3920 raft_consensus.cc:2463] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 2.
I20250111 02:53:05.095916 4724 raft_consensus.cc:988] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: : Instructing follower 8845a0bcc9d74fa19808e003237576f2 to start an election
I20250111 02:53:05.096071 4724 raft_consensus.cc:1076] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Signalling peer 8845a0bcc9d74fa19808e003237576f2 to start an election
I20250111 02:53:05.096539 4319 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb"
dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
from {username='slave'} at 127.0.0.1:44636
I20250111 02:53:05.096719 4319 raft_consensus.cc:491] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:05.096817 4319 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:05.099035 4319 raft_consensus.cc:513] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:05.099707 4319 leader_election.cc:290] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [CANDIDATE]: Term 4 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:53:05.100071 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "8845a0bcc9d74fa19808e003237576f2" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:05.100128 3920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08873d252254c8dbda59ce10b6766cb" candidate_uuid: "8845a0bcc9d74fa19808e003237576f2" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:53:05.100246 3839 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:05.100282 3920 raft_consensus.cc:3049] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Stepping down as leader of term 3
I20250111 02:53:05.100338 3920 raft_consensus.cc:738] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:05.100556 3920 consensus_queue.cc:260] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:05.100813 3920 raft_consensus.cc:3054] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Advancing to term 4
I20250111 02:53:05.102015 3839 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8845a0bcc9d74fa19808e003237576f2 in term 4.
I20250111 02:53:05.102332 3920 raft_consensus.cc:2463] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8845a0bcc9d74fa19808e003237576f2 in term 4.
I20250111 02:53:05.102363 4285 leader_election.cc:304] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8845a0bcc9d74fa19808e003237576f2, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:53:05.102612 4686 raft_consensus.cc:2798] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 4 FOLLOWER]: Leader election won for term 4
I20250111 02:53:05.103067 4686 raft_consensus.cc:695] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 4 LEADER]: Becoming Leader. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Running, Role: LEADER
I20250111 02:53:05.103291 4686 consensus_queue.cc:237] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:05.105471 3666 catalog_manager.cc:5526] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: term changed from 3 to 4, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6). New cstate: current_term: 4 leader_uuid: "8845a0bcc9d74fa19808e003237576f2" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:05.143698 4758 ts_tablet_manager.cc:927] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: Initiating tablet copy from peer 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:53:05.144356 4758 tablet_copy_client.cc:323] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:38147
I20250111 02:53:05.144836 3857 tablet_copy_service.cc:140] P 912e2307407d45dd88991584cd0b914d: Received BeginTabletCopySession request for tablet 7cf22853b8e742d98610db9dbcfea1c0 from peer a9e5558f33db487e97343f676589083e ({username='slave'} at 127.0.0.1:60352)
I20250111 02:53:05.144974 3857 tablet_copy_service.cc:161] P 912e2307407d45dd88991584cd0b914d: Beginning new tablet copy session on tablet 7cf22853b8e742d98610db9dbcfea1c0 from peer a9e5558f33db487e97343f676589083e at {username='slave'} at 127.0.0.1:60352: session id = a9e5558f33db487e97343f676589083e-7cf22853b8e742d98610db9dbcfea1c0
I20250111 02:53:05.147243 3857 tablet_copy_source_session.cc:215] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:05.148234 4758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf22853b8e742d98610db9dbcfea1c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:05.152632 4758 tablet_copy_client.cc:806] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:05.152854 4758 tablet_copy_client.cc:670] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:05.154253 4759 tablet_copy_client.cc:1025] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:05.184438 4760 ts_tablet_manager.cc:927] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: Initiating tablet copy from peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317)
I20250111 02:53:05.185176 4760 tablet_copy_client.cc:323] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.4:43317
I20250111 02:53:05.185709 4180 tablet_copy_service.cc:140] P a9e5558f33db487e97343f676589083e: Received BeginTabletCopySession request for tablet 6c67b2576c5f45b3b4e291de305ec91c from peer 5a90130f2bac4d13875c4094c826566d ({username='slave'} at 127.0.0.1:35312)
I20250111 02:53:05.185809 4180 tablet_copy_service.cc:161] P a9e5558f33db487e97343f676589083e: Beginning new tablet copy session on tablet 6c67b2576c5f45b3b4e291de305ec91c from peer 5a90130f2bac4d13875c4094c826566d at {username='slave'} at 127.0.0.1:35312: session id = 5a90130f2bac4d13875c4094c826566d-6c67b2576c5f45b3b4e291de305ec91c
I20250111 02:53:05.187781 4180 tablet_copy_source_session.cc:215] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:05.188771 4760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c67b2576c5f45b3b4e291de305ec91c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:05.189069 4762 ts_tablet_manager.cc:927] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: Initiating tablet copy from peer a9e5558f33db487e97343f676589083e (127.0.103.4:43317)
I20250111 02:53:05.189617 4762 tablet_copy_client.cc:323] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.4:43317
I20250111 02:53:05.190261 4180 tablet_copy_service.cc:140] P a9e5558f33db487e97343f676589083e: Received BeginTabletCopySession request for tablet 25a4915f163f405fa38c25cc03ceacc9 from peer 5a90130f2bac4d13875c4094c826566d ({username='slave'} at 127.0.0.1:35312)
I20250111 02:53:05.190376 4180 tablet_copy_service.cc:161] P a9e5558f33db487e97343f676589083e: Beginning new tablet copy session on tablet 25a4915f163f405fa38c25cc03ceacc9 from peer 5a90130f2bac4d13875c4094c826566d at {username='slave'} at 127.0.0.1:35312: session id = 5a90130f2bac4d13875c4094c826566d-25a4915f163f405fa38c25cc03ceacc9
I20250111 02:53:05.192314 4180 tablet_copy_source_session.cc:215] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:05.193293 4762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25a4915f163f405fa38c25cc03ceacc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:05.196460 4760 tablet_copy_client.cc:806] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:05.196660 4760 tablet_copy_client.cc:670] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:05.197716 4762 tablet_copy_client.cc:806] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:05.197978 4761 tablet_copy_client.cc:1025] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:05.197961 4762 tablet_copy_client.cc:670] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:05.199088 4763 tablet_copy_client.cc:1025] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:05.210060 4764 ts_tablet_manager.cc:927] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: Initiating tablet copy from peer 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147)
I20250111 02:53:05.210592 4764 tablet_copy_client.cc:323] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:38147
I20250111 02:53:05.211156 3857 tablet_copy_service.cc:140] P 912e2307407d45dd88991584cd0b914d: Received BeginTabletCopySession request for tablet 3bf60fc472f742518bd4328f9e88b436 from peer 8845a0bcc9d74fa19808e003237576f2 ({username='slave'} at 127.0.0.1:43602)
I20250111 02:53:05.211278 3857 tablet_copy_service.cc:161] P 912e2307407d45dd88991584cd0b914d: Beginning new tablet copy session on tablet 3bf60fc472f742518bd4328f9e88b436 from peer 8845a0bcc9d74fa19808e003237576f2 at {username='slave'} at 127.0.0.1:43602: session id = 8845a0bcc9d74fa19808e003237576f2-3bf60fc472f742518bd4328f9e88b436
I20250111 02:53:05.213430 3857 tablet_copy_source_session.cc:215] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:05.214352 4764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bf60fc472f742518bd4328f9e88b436. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:05.218727 4764 tablet_copy_client.cc:806] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:05.218938 4764 tablet_copy_client.cc:670] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:05.220253 4765 tablet_copy_client.cc:1025] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Injecting latency into file download: 2000
I20250111 02:53:05.525539 4245 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:53:05.526162 4657 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:05.535101 3920 raft_consensus.cc:1270] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250111 02:53:05.535720 4657 consensus_queue.cc:1035] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:05.624269 3920 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250111 02:53:05.624733 4686 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:05.637166 3839 raft_consensus.cc:1270] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250111 02:53:05.637758 4774 consensus_queue.cc:1035] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.002s
I20250111 02:53:06.927793 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:06.930639 3841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:53:06.930835 3841 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:53:06.931241 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:06.931429 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:07.057129 4751 tablet_copy_client.cc:538] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:07.059729 4751 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: Bootstrap starting.
I20250111 02:53:07.090206 4751 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:07.090541 4751 tablet_bootstrap.cc:492] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: Bootstrap complete.
I20250111 02:53:07.090775 4751 ts_tablet_manager.cc:1397] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: Time spent bootstrapping tablet: real 0.031s user 0.027s sys 0.005s
I20250111 02:53:07.091326 4751 raft_consensus.cc:357] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.091482 4751 raft_consensus.cc:738] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: a9e5558f33db487e97343f676589083e, State: Initialized, Role: LEARNER
I20250111 02:53:07.091634 4751 consensus_queue.cc:260] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 3.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.092509 4751 ts_tablet_manager.cc:1428] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:07.093050 3931 tablet_copy_service.cc:342] P 4667e917a8b44d0384ced54357111047: Request end of tablet copy session a9e5558f33db487e97343f676589083e-1e52445200d144479f643c05ecb3a0f8 received from {username='slave'} at 127.0.0.1:36490
I20250111 02:53:07.093151 3931 tablet_copy_service.cc:434] P 4667e917a8b44d0384ced54357111047: ending tablet copy session a9e5558f33db487e97343f676589083e-1e52445200d144479f643c05ecb3a0f8 on tablet 1e52445200d144479f643c05ecb3a0f8 with peer a9e5558f33db487e97343f676589083e
I20250111 02:53:07.154595 4758 tablet_copy_client.cc:538] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:07.157328 4758 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: Bootstrap starting.
I20250111 02:53:07.192453 4758 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=119 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:07.192812 4758 tablet_bootstrap.cc:492] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: Bootstrap complete.
I20250111 02:53:07.193030 4758 ts_tablet_manager.cc:1397] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: Time spent bootstrapping tablet: real 0.036s user 0.035s sys 0.000s
I20250111 02:53:07.193614 4758 raft_consensus.cc:357] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.193835 4758 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: a9e5558f33db487e97343f676589083e, State: Initialized, Role: LEARNER
I20250111 02:53:07.194031 4758 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.194473 4758 ts_tablet_manager.cc:1428] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20250111 02:53:07.195071 3857 tablet_copy_service.cc:342] P 912e2307407d45dd88991584cd0b914d: Request end of tablet copy session a9e5558f33db487e97343f676589083e-7cf22853b8e742d98610db9dbcfea1c0 received from {username='slave'} at 127.0.0.1:60352
I20250111 02:53:07.195176 3857 tablet_copy_service.cc:434] P 912e2307407d45dd88991584cd0b914d: ending tablet copy session a9e5558f33db487e97343f676589083e-7cf22853b8e742d98610db9dbcfea1c0 on tablet 7cf22853b8e742d98610db9dbcfea1c0 with peer a9e5558f33db487e97343f676589083e
I20250111 02:53:07.198278 4760 tablet_copy_client.cc:538] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:07.198856 4786 raft_consensus.cc:988] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:07.198962 4786 raft_consensus.cc:1076] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:07.199389 4762 tablet_copy_client.cc:538] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:07.199427 3772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44944
I20250111 02:53:07.199611 3772 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:07.199676 3772 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:53:07.201296 3772 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.201591 4760 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: Bootstrap starting.
I20250111 02:53:07.201941 3772 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 6 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:07.202250 4762 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: Bootstrap starting.
I20250111 02:53:07.202448 4319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:07.202600 4319 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:53:07.203608 3839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:07.203779 3839 raft_consensus.cc:3049] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Stepping down as leader of term 5
I20250111 02:53:07.203830 3839 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:07.204051 3839 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.204421 3839 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:53:07.204872 4319 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 6.
I20250111 02:53:07.205237 3737 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 8845a0bcc9d74fa19808e003237576f2; no voters:
I20250111 02:53:07.205472 4712 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 FOLLOWER]: Leader election won for term 6
I20250111 02:53:07.206835 3839 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 6.
I20250111 02:53:07.211203 4712 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:07.211530 4712 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.215162 3666 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 5 to 6, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 6 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:07.220606 4764 tablet_copy_client.cc:538] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:07.223903 4764 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap starting.
I20250111 02:53:07.237550 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:07.237749 3772 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 LEADER]: Received request to transfer leadership
I20250111 02:53:07.245649 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:07.245993 3772 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 LEADER]: Received request to transfer leadership
W20250111 02:53:07.246345 3654 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 7cf22853b8e742d98610db9dbcfea1c0 already in progress [suppressed 1 similar messages]
I20250111 02:53:07.254187 4762 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:07.254755 4762 tablet_bootstrap.cc:492] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: Bootstrap complete.
I20250111 02:53:07.255072 4762 ts_tablet_manager.cc:1397] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: Time spent bootstrapping tablet: real 0.053s user 0.042s sys 0.001s
I20250111 02:53:07.256112 4762 raft_consensus.cc:357] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:07.256348 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:07.256402 4762 raft_consensus.cc:738] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5a90130f2bac4d13875c4094c826566d, State: Initialized, Role: LEARNER
I20250111 02:53:07.256491 3772 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 LEADER]: Received request to transfer leadership
I20250111 02:53:07.256579 4762 consensus_queue.cc:260] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
W20250111 02:53:07.256947 3718 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 7cf22853b8e742d98610db9dbcfea1c0 already in progress
W20250111 02:53:07.257015 3718 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 7cf22853b8e742d98610db9dbcfea1c0 already in progress
I20250111 02:53:07.258833 4762 ts_tablet_manager.cc:1428] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:53:07.259810 4180 tablet_copy_service.cc:342] P a9e5558f33db487e97343f676589083e: Request end of tablet copy session 5a90130f2bac4d13875c4094c826566d-25a4915f163f405fa38c25cc03ceacc9 received from {username='slave'} at 127.0.0.1:35312
I20250111 02:53:07.259903 4180 tablet_copy_service.cc:434] P a9e5558f33db487e97343f676589083e: ending tablet copy session 5a90130f2bac4d13875c4094c826566d-25a4915f163f405fa38c25cc03ceacc9 on tablet 25a4915f163f405fa38c25cc03ceacc9 with peer 5a90130f2bac4d13875c4094c826566d
I20250111 02:53:07.260666 4760 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:07.261003 4170 raft_consensus.cc:1212] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 LEARNER]: Deduplicated request from leader. Original: 3.32->[3.33-3.33] Dedup: 3.33->[]
I20250111 02:53:07.261098 4760 tablet_bootstrap.cc:492] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: Bootstrap complete.
I20250111 02:53:07.261332 4760 ts_tablet_manager.cc:1397] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: Time spent bootstrapping tablet: real 0.060s user 0.044s sys 0.004s
I20250111 02:53:07.262712 4760 raft_consensus.cc:357] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:07.263121 4760 raft_consensus.cc:738] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5a90130f2bac4d13875c4094c826566d, State: Initialized, Role: LEARNER
I20250111 02:53:07.263305 4760 consensus_queue.cc:260] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 2.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: NON_VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: true } }
I20250111 02:53:07.263808 4760 ts_tablet_manager.cc:1428] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250111 02:53:07.264509 4180 tablet_copy_service.cc:342] P a9e5558f33db487e97343f676589083e: Request end of tablet copy session 5a90130f2bac4d13875c4094c826566d-6c67b2576c5f45b3b4e291de305ec91c received from {username='slave'} at 127.0.0.1:35312
I20250111 02:53:07.264622 4180 tablet_copy_service.cc:434] P a9e5558f33db487e97343f676589083e: ending tablet copy session 5a90130f2bac4d13875c4094c826566d-6c67b2576c5f45b3b4e291de305ec91c on tablet 6c67b2576c5f45b3b4e291de305ec91c with peer 5a90130f2bac4d13875c4094c826566d
I20250111 02:53:07.284554 4764 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=120 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:07.284971 4764 tablet_bootstrap.cc:492] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: Bootstrap complete.
I20250111 02:53:07.285231 4764 ts_tablet_manager.cc:1397] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: Time spent bootstrapping tablet: real 0.061s user 0.039s sys 0.008s
I20250111 02:53:07.285935 4764 raft_consensus.cc:357] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:07.286191 4764 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Initialized, Role: LEARNER
I20250111 02:53:07.286384 4764 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 4.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: NON_VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: true } }
I20250111 02:53:07.286857 4764 ts_tablet_manager.cc:1428] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250111 02:53:07.287462 3857 tablet_copy_service.cc:342] P 912e2307407d45dd88991584cd0b914d: Request end of tablet copy session 8845a0bcc9d74fa19808e003237576f2-3bf60fc472f742518bd4328f9e88b436 received from {username='slave'} at 127.0.0.1:43602
I20250111 02:53:07.287586 3857 tablet_copy_service.cc:434] P 912e2307407d45dd88991584cd0b914d: ending tablet copy session 8845a0bcc9d74fa19808e003237576f2-3bf60fc472f742518bd4328f9e88b436 on tablet 3bf60fc472f742518bd4328f9e88b436 with peer 8845a0bcc9d74fa19808e003237576f2
I20250111 02:53:07.329108 4319 raft_consensus.cc:1212] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.30->[4.31-4.31] Dedup: 4.31->[]
I20250111 02:53:07.484836 4245 raft_consensus.cc:1212] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.32->[2.33-2.33] Dedup: 2.33->[]
I20250111 02:53:07.616146 4170 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 5 LEARNER]: Advancing to term 6
I20250111 02:53:07.618500 4170 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 6 LEARNER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250111 02:53:07.618975 4712 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250111 02:53:07.621241 4712 raft_consensus.cc:1059] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: attempting to promote NON_VOTER a9e5558f33db487e97343f676589083e to VOTER
W20250111 02:53:07.621388 4712 raft_consensus.cc:1063] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: Unable to promote non-voter a9e5558f33db487e97343f676589083e: Service unavailable: leader transfer in progress
I20250111 02:53:07.625128 3841 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 6 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250111 02:53:07.625639 4787 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250111 02:53:07.627830 4712 raft_consensus.cc:988] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:07.627980 4712 raft_consensus.cc:1076] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:07.628504 3839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60314
I20250111 02:53:07.629781 3839 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:07.629791 4319 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 6 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250111 02:53:07.629979 3839 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:53:07.630512 4787 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250111 02:53:07.632645 3839 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.634195 4318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 7 candidate_status { last_received { term: 6 index: 35 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:07.634537 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 7 candidate_status { last_received { term: 6 index: 35 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:07.635555 3839 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 7 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:07.641520 3772 raft_consensus.cc:3049] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 LEADER]: Stepping down as leader of term 6
I20250111 02:53:07.641629 3772 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:07.642017 3772 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 6.35, Last appended by leader: 35, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.642385 3772 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:53:07.644587 3772 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 7.
I20250111 02:53:07.645136 3810 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 912e2307407d45dd88991584cd0b914d; no voters: 8845a0bcc9d74fa19808e003237576f2
I20250111 02:53:07.645397 4786 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 FOLLOWER]: Leader election won for term 7
I20250111 02:53:07.645534 4786 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:07.645807 4786 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 6.35, Last appended by leader: 35, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } }
I20250111 02:53:07.649163 3670 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 6 to 7, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 7 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:07.665779 4245 raft_consensus.cc:1212] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Deduplicated request from leader. Original: 2.31->[2.32-2.32] Dedup: 2.32->[]
I20250111 02:53:07.719179 4805 raft_consensus.cc:1059] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: attempting to promote NON_VOTER a9e5558f33db487e97343f676589083e to VOTER
I20250111 02:53:07.720381 4805 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:07.722448 4170 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 LEARNER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250111 02:53:07.722798 3839 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250111 02:53:07.722904 4245 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250111 02:53:07.722970 4782 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:07.723320 4805 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:07.723644 4808 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:07.726833 4806 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:07.727290 4245 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:07.729754 3841 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:07.731169 3668 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d reported cstate change: config changed from index 33 to 34, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:07.730015 4170 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:07.735451 3916 consensus_queue.cc:237] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 3.34, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:07.737161 4245 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20250111 02:53:07.737519 4169 raft_consensus.cc:1270] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 3 index: 34. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20250111 02:53:07.738044 4805 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250111 02:53:07.738312 4782 consensus_queue.cc:1035] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250111 02:53:07.740509 4808 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:07.740970 4170 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:07.743350 3654 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 1e52445200d144479f643c05ecb3a0f8 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:07.744714 3666 catalog_manager.cc:5526] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 reported cstate change: config changed from index 34 to 35, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New cstate: current_term: 3 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:07.746053 4245 raft_consensus.cc:2949] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Committing config change with OpId 3.35: config changed from index 34 to 35, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:07.747949 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e52445200d144479f643c05ecb3a0f8 with delete_type TABLET_DATA_TOMBSTONED (TS 912e2307407d45dd88991584cd0b914d not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:60308
I20250111 02:53:07.760329 4812 tablet_replica.cc:331] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:07.760565 4812 raft_consensus.cc:2238] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:07.760854 4812 raft_consensus.cc:2267] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:07.761620 4812 ts_tablet_manager.cc:1905] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:07.765638 4812 ts_tablet_manager.cc:1918] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.34
I20250111 02:53:07.765748 4812 log.cc:1198] T 1e52445200d144479f643c05ecb3a0f8 P 912e2307407d45dd88991584cd0b914d: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root/wals/1e52445200d144479f643c05ecb3a0f8
I20250111 02:53:07.777635 3657 catalog_manager.cc:4872] TS 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147): tablet 1e52445200d144479f643c05ecb3a0f8 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:07.943341 4657 raft_consensus.cc:1059] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: attempting to promote NON_VOTER 8845a0bcc9d74fa19808e003237576f2 to VOTER
I20250111 02:53:07.944335 4657 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 4.31, Last appended by leader: 29, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:07.955387 4319 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 4 LEARNER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250111 02:53:07.956034 4786 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:07.956451 3916 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250111 02:53:07.959095 4814 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:07.962535 3772 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 4 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250111 02:53:07.963546 4657 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250111 02:53:07.970155 4786 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 LEADER]: Committing config change with OpId 4.32: config changed from index 31 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:07.975720 3916 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:07.976106 3772 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:07.981491 3670 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: config changed from index 31 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:07.987080 4319 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 4 FOLLOWER]: Committing config change with OpId 4.32: config changed from index 31 to 32, 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } attrs { replace: true } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:07.994995 3841 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:08.000278 3772 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250111 02:53:08.000298 4318 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 4 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250111 02:53:08.001183 4818 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:08.001608 4657 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:08.006273 4814 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:08.010659 3657 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 3bf60fc472f742518bd4328f9e88b436 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:08.010879 4318 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:08.014570 3772 raft_consensus.cc:2949] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 32 to 33, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } }
I20250111 02:53:08.016304 3666 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index 32 to 33, VOTER 4667e917a8b44d0384ced54357111047 (127.0.103.3) evicted. New cstate: current_term: 4 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:08.027384 3909 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bf60fc472f742518bd4328f9e88b436 with delete_type TABLET_DATA_TOMBSTONED (TS 4667e917a8b44d0384ced54357111047 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:58806
I20250111 02:53:08.050591 4819 raft_consensus.cc:1059] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: attempting to promote NON_VOTER 5a90130f2bac4d13875c4094c826566d to VOTER
I20250111 02:53:08.051258 4820 tablet_replica.cc:331] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:08.053100 4820 raft_consensus.cc:2238] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:08.053464 4820 raft_consensus.cc:2267] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:08.055466 4820 ts_tablet_manager.cc:1905] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:08.056270 4819 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:08.065549 4820 ts_tablet_manager.cc:1918] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 4.32
I20250111 02:53:08.065703 4820 log.cc:1198] T 3bf60fc472f742518bd4328f9e88b436 P 4667e917a8b44d0384ced54357111047: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-2-root/wals/3bf60fc472f742518bd4328f9e88b436
I20250111 02:53:08.066511 3654 catalog_manager.cc:4872] TS 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021): tablet 3bf60fc472f742518bd4328f9e88b436 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:08.071338 4245 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:08.072016 4727 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:08.076447 4318 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:53:08.080159 4318 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20250111 02:53:08.080832 4818 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250111 02:53:08.087440 3772 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20250111 02:53:08.088184 4170 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 6 LEARNER]: Advancing to term 7
I20250111 02:53:08.088122 4657 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250111 02:53:08.096733 4170 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 7 LEARNER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20250111 02:53:08.099889 4657 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250111 02:53:08.102459 4657 raft_consensus.cc:1059] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: attempting to promote NON_VOTER a9e5558f33db487e97343f676589083e to VOTER
I20250111 02:53:08.103397 4657 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 7.36, Last appended by leader: 35, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:08.104882 4318 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:08.105417 4727 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:08.105677 4168 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 7 LEARNER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 7 index: 36. Preceding OpId from leader: term: 7 index: 37. (index mismatch)
I20250111 02:53:08.106231 4786 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250111 02:53:08.109184 4318 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 7 index: 36. Preceding OpId from leader: term: 7 index: 37. (index mismatch)
I20250111 02:53:08.109602 3772 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 7 index: 36. Preceding OpId from leader: term: 7 index: 37. (index mismatch)
I20250111 02:53:08.109838 4786 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250111 02:53:08.110194 4657 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250111 02:53:08.113179 3772 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Committing config change with OpId 7.37: config changed from index 34 to 37, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:08.113210 4317 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Committing config change with OpId 7.37: config changed from index 34 to 37, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:08.114799 3770 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:08.115358 4819 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:08.112389 4657 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 LEADER]: Committing config change with OpId 7.37: config changed from index 34 to 37, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:08.122028 3666 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 34 to 37, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:08.131683 4168 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 7 FOLLOWER]: Committing config change with OpId 7.37: config changed from index 34 to 37, a9e5558f33db487e97343f676589083e (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:08.140460 3841 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 7.37, Last appended by leader: 35, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:08.142879 4317 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 7 index: 37. Preceding OpId from leader: term: 7 index: 38. (index mismatch)
I20250111 02:53:08.143479 4818 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20250111 02:53:08.145385 4727 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.148178 4818 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 LEADER]: Committing config change with OpId 7.38: config changed from index 37 to 38, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:08.150490 3770 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.153156 4317 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Committing config change with OpId 7.38: config changed from index 37 to 38, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:08.155830 4319 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.157361 3657 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7cf22853b8e742d98610db9dbcfea1c0 with cas_config_opid_index 37: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:08.161731 3670 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: config changed from index 37 to 38, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New cstate: current_term: 7 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:08.168205 4169 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 7 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 7 index: 37. Preceding OpId from leader: term: 7 index: 38. (index mismatch)
I20250111 02:53:08.171240 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf22853b8e742d98610db9dbcfea1c0 with delete_type TABLET_DATA_TOMBSTONED (TS 18a7cd9ed0644218857e3da957957cf8 not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:08.170902 3668 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 33 to 34, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.171942 4657 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20250111 02:53:08.172712 4826 tablet_replica.cc:331] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:08.174139 4826 raft_consensus.cc:2238] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:08.175158 4170 raft_consensus.cc:2949] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 7 FOLLOWER]: Committing config change with OpId 7.38: config changed from index 37 to 38, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } }
I20250111 02:53:08.183861 4826 raft_consensus.cc:2267] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:08.185487 4826 ts_tablet_manager.cc:1905] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:08.194828 4169 consensus_queue.cc:237] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 2.34, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:08.196560 4821 raft_consensus.cc:1059] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: attempting to promote NON_VOTER 5a90130f2bac4d13875c4094c826566d to VOTER
I20250111 02:53:08.197532 4821 consensus_queue.cc:237] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 30, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:08.199927 4826 ts_tablet_manager.cc:1918] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.37
I20250111 02:53:08.200134 4826 log.cc:1198] T 7cf22853b8e742d98610db9dbcfea1c0 P 18a7cd9ed0644218857e3da957957cf8: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root/wals/7cf22853b8e742d98610db9dbcfea1c0
I20250111 02:53:08.200762 4244 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20250111 02:53:08.201167 3654 catalog_manager.cc:4872] TS 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435): tablet 7cf22853b8e742d98610db9dbcfea1c0 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:08.202955 4319 raft_consensus.cc:1270] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 34. Preceding OpId from leader: term: 2 index: 35. (index mismatch)
I20250111 02:53:08.202503 4727 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 33, Time since last communication: 0.001s
I20250111 02:53:08.203905 4727 consensus_queue.cc:1035] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250111 02:53:08.204910 4244 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.213940 4317 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.214641 4243 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 LEARNER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:08.215291 4819 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:08.218654 3841 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:08.219964 3666 catalog_manager.cc:5526] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: config changed from index 34 to 35, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.212960 4727 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.222582 4819 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:08.235347 4244 raft_consensus.cc:2949] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Committing config change with OpId 2.35: config changed from index 34 to 35, VOTER 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) evicted. New config: { opid_index: 35 OBSOLETE_local: false peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.250581 3715 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 25a4915f163f405fa38c25cc03ceacc9 with cas_config_opid_index 34: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:08.258252 3754 tablet_service.cc:1514] Processing DeleteTablet for tablet 25a4915f163f405fa38c25cc03ceacc9 with delete_type TABLET_DATA_TOMBSTONED (TS 18a7cd9ed0644218857e3da957957cf8 not found in new config with opid_index 35) from {username='slave'} at 127.0.0.1:44914
I20250111 02:53:08.258792 4826 tablet_replica.cc:331] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:08.259049 4826 raft_consensus.cc:2238] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:08.259430 4826 raft_consensus.cc:2267] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:08.261497 4826 ts_tablet_manager.cc:1905] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:08.270830 4830 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.271624 4244 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.274286 4826 ts_tablet_manager.cc:1918] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.34
I20250111 02:53:08.274500 4826 log.cc:1198] T 25a4915f163f405fa38c25cc03ceacc9 P 18a7cd9ed0644218857e3da957957cf8: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-0-root/wals/25a4915f163f405fa38c25cc03ceacc9
I20250111 02:53:08.275312 3654 catalog_manager.cc:4872] TS 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435): tablet 25a4915f163f405fa38c25cc03ceacc9 (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:08.278049 3666 catalog_manager.cc:5526] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d reported cstate change: config changed from index 32 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.283922 3770 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250111 02:53:08.285472 4819 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250111 02:53:08.288827 3841 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.307837 3772 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, 5a90130f2bac4d13875c4094c826566d (127.0.103.5) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } attrs { replace: true } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.675644 4170 consensus_queue.cc:237] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } }
I20250111 02:53:08.677562 4244 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:08.678023 3772 raft_consensus.cc:1270] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer a9e5558f33db487e97343f676589083e: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 2 index: 34. (index mismatch)
I20250111 02:53:08.678275 4828 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:08.679059 4828 consensus_queue.cc:1035] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:08.680305 4819 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.681020 4244 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.681658 3772 raft_consensus.cc:2949] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Committing config change with OpId 2.34: config changed from index 33 to 34, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.685200 3715 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 6c67b2576c5f45b3b4e291de305ec91c with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:08.685505 3668 catalog_manager.cc:5526] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d reported cstate change: config changed from index 33 to 34, VOTER 912e2307407d45dd88991584cd0b914d (127.0.103.2) evicted. New cstate: current_term: 2 leader_uuid: "a9e5558f33db487e97343f676589083e" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } } peers { permanent_uuid: "5a90130f2bac4d13875c4094c826566d" member_type: VOTER last_known_addr { host: "127.0.103.5" port: 41295 } attrs { promote: false } } }
I20250111 02:53:08.688351 3833 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c67b2576c5f45b3b4e291de305ec91c with delete_type TABLET_DATA_TOMBSTONED (TS 912e2307407d45dd88991584cd0b914d not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:60308
I20250111 02:53:08.689033 4833 tablet_replica.cc:331] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:08.689177 4833 raft_consensus.cc:2238] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:08.689366 4833 raft_consensus.cc:2267] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:08.690030 4833 ts_tablet_manager.cc:1905] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:08.693622 4833 ts_tablet_manager.cc:1918] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.33
I20250111 02:53:08.693717 4833 log.cc:1198] T 6c67b2576c5f45b3b4e291de305ec91c P 912e2307407d45dd88991584cd0b914d: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0/minicluster-data/ts-1-root/wals/6c67b2576c5f45b3b4e291de305ec91c
I20250111 02:53:08.694311 3657 catalog_manager.cc:4872] TS 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147): tablet 6c67b2576c5f45b3b4e291de305ec91c (table test-workload [id=913eeb487ba14c7c9f287fff752a835e]) successfully deleted
I20250111 02:53:08.931859 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:08.934506 3841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:53:08.934710 3841 raft_consensus.cc:604] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:53:08.935216 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:08.935420 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:09.310266 4818 raft_consensus.cc:988] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:09.310423 4818 raft_consensus.cc:1076] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:09.311012 3772 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44944
I20250111 02:53:09.311216 3772 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:09.311319 3772 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:09.313035 3772 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:09.313596 3772 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 5 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:09.314016 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:09.314061 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:09.314244 3841 raft_consensus.cc:3049] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 LEADER]: Stepping down as leader of term 4
I20250111 02:53:09.314270 4317 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:09.314352 3841 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:09.314644 3841 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:09.314986 3841 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:09.316856 4317 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 5.
I20250111 02:53:09.317296 3841 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 5.
I20250111 02:53:09.317296 3737 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 8845a0bcc9d74fa19808e003237576f2; no voters:
I20250111 02:53:09.317600 4801 raft_consensus.cc:2798] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Leader election won for term 5
I20250111 02:53:09.318037 4801 raft_consensus.cc:695] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 5 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:09.318286 4801 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:09.320488 3670 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 4 to 5, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 5 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:09.709802 4317 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250111 02:53:09.710361 4801 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:09.713948 3841 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250111 02:53:09.714452 4801 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250111 02:53:10.935845 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:10.938544 3772 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:53:10.938738 3772 raft_consensus.cc:604] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 5 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:53:10.939200 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:10.939400 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:11.160013 4847 raft_consensus.cc:988] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:11.160151 4847 raft_consensus.cc:1076] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 5 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:11.160681 3841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60314
I20250111 02:53:11.160859 3841 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:11.160936 3841 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:53:11.162703 3841 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:11.163312 3841 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 6 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:11.163686 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:11.163640 3772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:11.163827 4317 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:53:11.163827 3772 raft_consensus.cc:3049] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 5 LEADER]: Stepping down as leader of term 5
I20250111 02:53:11.163937 3772 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:11.164173 3772 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:11.164500 3772 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Advancing to term 6
I20250111 02:53:11.166319 4317 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 6.
I20250111 02:53:11.166703 3812 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8845a0bcc9d74fa19808e003237576f2, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:53:11.166780 3772 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 6.
I20250111 02:53:11.166996 4818 raft_consensus.cc:2798] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 6 FOLLOWER]: Leader election won for term 6
I20250111 02:53:11.167390 4818 raft_consensus.cc:695] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 6 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:11.167606 4818 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:11.169889 3666 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 5 to 6, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 6 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:11.620976 4317 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 6 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250111 02:53:11.621462 4818 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250111 02:53:11.626183 3772 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 6 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250111 02:53:11.626667 4848 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250111 02:53:12.939913 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:12.942471 3841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "8845a0bcc9d74fa19808e003237576f2"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:53:12.942690 3841 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 LEADER]: Received request to transfer leadership to TS 8845a0bcc9d74fa19808e003237576f2
I20250111 02:53:12.943146 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:12.943367 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:13.390094 4860 raft_consensus.cc:988] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 8845a0bcc9d74fa19808e003237576f2 to start an election
I20250111 02:53:13.390254 4860 raft_consensus.cc:1076] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 LEADER]: Signalling peer 8845a0bcc9d74fa19808e003237576f2 to start an election
I20250111 02:53:13.390816 4317 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
from {username='slave'} at 127.0.0.1:45048
I20250111 02:53:13.391008 4317 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:13.391131 4317 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Advancing to term 8
I20250111 02:53:13.392799 4317 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:13.393535 4317 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [CANDIDATE]: Term 8 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), a9e5558f33db487e97343f676589083e (127.0.103.4:43317)
I20250111 02:53:13.393699 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "8845a0bcc9d74fa19808e003237576f2" candidate_term: 8 candidate_status { last_received { term: 7 index: 38 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:13.393829 4169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "8845a0bcc9d74fa19808e003237576f2" candidate_term: 8 candidate_status { last_received { term: 7 index: 38 } } ignore_live_leader: true dest_uuid: "a9e5558f33db487e97343f676589083e"
I20250111 02:53:13.394057 3841 raft_consensus.cc:3049] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 LEADER]: Stepping down as leader of term 7
I20250111 02:53:13.394066 4169 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 7 FOLLOWER]: Advancing to term 8
I20250111 02:53:13.394163 3841 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:13.394387 3841 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 7.38, Last appended by leader: 38, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:13.394670 3841 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 7 FOLLOWER]: Advancing to term 8
I20250111 02:53:13.395961 4169 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8845a0bcc9d74fa19808e003237576f2 in term 8.
I20250111 02:53:13.396325 4283 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8845a0bcc9d74fa19808e003237576f2, a9e5558f33db487e97343f676589083e; no voters:
I20250111 02:53:13.396386 3841 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8845a0bcc9d74fa19808e003237576f2 in term 8.
I20250111 02:53:13.396602 4776 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 8 FOLLOWER]: Leader election won for term 8
I20250111 02:53:13.397008 4776 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 8 LEADER]: Becoming Leader. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Running, Role: LEADER
I20250111 02:53:13.397261 4776 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 7.38, Last appended by leader: 38, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:13.399263 3666 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 reported cstate change: term changed from 7 to 8, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6). New cstate: current_term: 8 leader_uuid: "8845a0bcc9d74fa19808e003237576f2" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:13.851504 4169 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 8 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 7 index: 38. Preceding OpId from leader: term: 8 index: 39. (index mismatch)
I20250111 02:53:13.852159 4776 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20250111 02:53:13.856040 3841 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 8 FOLLOWER]: Refusing update from remote peer 8845a0bcc9d74fa19808e003237576f2: Log matching property violated. Preceding OpId in replica: term: 7 index: 38. Preceding OpId from leader: term: 8 index: 39. (index mismatch)
I20250111 02:53:13.856714 4861 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20250111 02:53:14.943825 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:14.951288 4317 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:54940
I20250111 02:53:14.951473 4317 raft_consensus.cc:604] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 8 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:53:14.951886 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:14.952091 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:15.492173 4874 raft_consensus.cc:988] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:15.492331 4874 raft_consensus.cc:1076] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 8 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:15.492864 3841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:43602
I20250111 02:53:15.493046 3841 raft_consensus.cc:491] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:15.493158 3841 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 8 FOLLOWER]: Advancing to term 9
I20250111 02:53:15.494923 3841 raft_consensus.cc:513] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:15.495584 3841 leader_election.cc:290] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 9 election: Requested vote from peers 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871), a9e5558f33db487e97343f676589083e (127.0.103.4:43317)
I20250111 02:53:15.495934 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 9 candidate_status { last_received { term: 8 index: 39 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:15.496038 4169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf22853b8e742d98610db9dbcfea1c0" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 9 candidate_status { last_received { term: 8 index: 39 } } ignore_live_leader: true dest_uuid: "a9e5558f33db487e97343f676589083e"
I20250111 02:53:15.496131 4317 raft_consensus.cc:3049] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 8 LEADER]: Stepping down as leader of term 8
I20250111 02:53:15.496177 4169 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 8 FOLLOWER]: Advancing to term 9
I20250111 02:53:15.496228 4317 raft_consensus.cc:738] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 8845a0bcc9d74fa19808e003237576f2, State: Running, Role: LEADER
I20250111 02:53:15.496488 4317 consensus_queue.cc:260] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 8.39, Last appended by leader: 39, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:15.496831 4317 raft_consensus.cc:3054] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 8 FOLLOWER]: Advancing to term 9
I20250111 02:53:15.498771 4169 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 9.
I20250111 02:53:15.499161 3811 leader_election.cc:304] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 912e2307407d45dd88991584cd0b914d, a9e5558f33db487e97343f676589083e; no voters:
I20250111 02:53:15.499280 4317 raft_consensus.cc:2463] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 9.
I20250111 02:53:15.499886 4860 raft_consensus.cc:2798] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 9 FOLLOWER]: Leader election won for term 9
I20250111 02:53:15.500285 4860 raft_consensus.cc:695] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 9 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:15.500545 4860 consensus_queue.cc:237] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 8.39, Last appended by leader: 39, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } }
I20250111 02:53:15.502799 3666 catalog_manager.cc:5526] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 8 to 9, leader changed from 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 9 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:16.026753 4317 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 9 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 8 index: 39. Preceding OpId from leader: term: 9 index: 40. (index mismatch)
I20250111 02:53:16.027283 4860 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250111 02:53:16.031172 4169 raft_consensus.cc:1270] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 9 FOLLOWER]: Refusing update from remote peer 912e2307407d45dd88991584cd0b914d: Log matching property violated. Preceding OpId in replica: term: 8 index: 39. Preceding OpId from leader: term: 9 index: 40. (index mismatch)
I20250111 02:53:16.031754 4860 consensus_queue.cc:1035] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e5558f33db487e97343f676589083e" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 43317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250111 02:53:16.952505 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:16.955315 3841 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:43596
I20250111 02:53:16.955543 3841 raft_consensus.cc:604] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 6 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:53:16.956019 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:16.956238 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:17.313637 4893 raft_consensus.cc:988] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:17.313906 4893 raft_consensus.cc:1076] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 6 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:17.314469 3770 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44944
I20250111 02:53:17.314673 3770 raft_consensus.cc:491] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:17.314780 3770 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:53:17.316671 3770 raft_consensus.cc:513] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:17.317219 3770 leader_election.cc:290] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 7 election: Requested vote from peers 912e2307407d45dd88991584cd0b914d (127.0.103.2:38147), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:17.317605 4317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 7 candidate_status { last_received { term: 6 index: 35 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:17.317605 3841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bf60fc472f742518bd4328f9e88b436" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 7 candidate_status { last_received { term: 6 index: 35 } } ignore_live_leader: true dest_uuid: "912e2307407d45dd88991584cd0b914d"
I20250111 02:53:17.317873 3841 raft_consensus.cc:3049] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 6 LEADER]: Stepping down as leader of term 6
I20250111 02:53:17.317885 4317 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:53:17.317987 3841 raft_consensus.cc:738] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:17.318255 3841 consensus_queue.cc:260] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 6.35, Last appended by leader: 35, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:17.318610 3841 raft_consensus.cc:3054] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 6 FOLLOWER]: Advancing to term 7
I20250111 02:53:17.320578 4317 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 7.
I20250111 02:53:17.321002 3841 raft_consensus.cc:2463] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 7.
I20250111 02:53:17.321067 3737 leader_election.cc:304] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 8845a0bcc9d74fa19808e003237576f2; no voters:
I20250111 02:53:17.321367 4847 raft_consensus.cc:2798] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Leader election won for term 7
I20250111 02:53:17.321743 4847 raft_consensus.cc:695] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 7 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:17.322019 4847 consensus_queue.cc:237] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 6.35, Last appended by leader: 35, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:17.324468 3670 catalog_manager.cc:5526] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 6 to 7, leader changed from 912e2307407d45dd88991584cd0b914d (127.0.103.2) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 7 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:17.751633 4317 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20250111 02:53:17.752214 4847 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250111 02:53:17.760247 3841 raft_consensus.cc:1270] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 7 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20250111 02:53:17.761056 4903 consensus_queue.cc:1035] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250111 02:53:18.956647 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:18.959415 3770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:53:18.959635 3770 raft_consensus.cc:604] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Received request to transfer leadership to TS 4667e917a8b44d0384ced54357111047
I20250111 02:53:18.960106 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:18.960307 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:19.172986 4907 raft_consensus.cc:988] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:53:19.173156 4907 raft_consensus.cc:1076] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Signalling peer 4667e917a8b44d0384ced54357111047 to start an election
I20250111 02:53:19.173667 3916 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4"
dest_uuid: "4667e917a8b44d0384ced54357111047"
from {username='slave'} at 127.0.0.1:58812
I20250111 02:53:19.173885 3916 raft_consensus.cc:491] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:19.174001 3916 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:19.176420 3916 raft_consensus.cc:513] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:19.177312 3916 leader_election.cc:290] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 2 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:19.177673 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:19.177904 3770 raft_consensus.cc:3049] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:53:19.177870 4319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4" candidate_uuid: "4667e917a8b44d0384ced54357111047" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:19.177980 3770 raft_consensus.cc:738] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:19.178050 4319 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:19.178198 3770 consensus_queue.cc:260] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:19.178437 3770 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:19.179689 4319 raft_consensus.cc:2463] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 2.
I20250111 02:53:19.180047 3770 raft_consensus.cc:2463] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4667e917a8b44d0384ced54357111047 in term 2.
I20250111 02:53:19.180078 3886 leader_election.cc:304] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 8845a0bcc9d74fa19808e003237576f2; no voters:
I20250111 02:53:19.180344 4782 raft_consensus.cc:2798] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:53:19.180730 4782 raft_consensus.cc:695] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Becoming Leader. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:19.180981 4782 consensus_queue.cc:237] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:19.183192 3670 catalog_manager.cc:5526] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 reported cstate change: term changed from 1 to 2, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 4667e917a8b44d0384ced54357111047 (127.0.103.3). New cstate: current_term: 2 leader_uuid: "4667e917a8b44d0384ced54357111047" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:19.713511 4319 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250111 02:53:19.714044 4782 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:19.718009 3770 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Refusing update from remote peer 4667e917a8b44d0384ced54357111047: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250111 02:53:19.718564 4916 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250111 02:53:20.960903 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:20.964102 3916 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4"
dest_uuid: "4667e917a8b44d0384ced54357111047"
mode: GRACEFUL
new_leader_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:48134
I20250111 02:53:20.964370 3916 raft_consensus.cc:604] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Received request to transfer leadership to TS 18a7cd9ed0644218857e3da957957cf8
I20250111 02:53:20.964902 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:20.965114 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:21.119964 4919 raft_consensus.cc:988] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047: : Instructing follower 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:21.120136 4919 raft_consensus.cc:1076] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Signalling peer 18a7cd9ed0644218857e3da957957cf8 to start an election
I20250111 02:53:21.120865 3770 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
from {username='slave'} at 127.0.0.1:44928
I20250111 02:53:21.121053 3770 raft_consensus.cc:491] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:21.121167 3770 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:53:21.122969 3770 raft_consensus.cc:513] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:21.123708 3770 leader_election.cc:290] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 3 election: Requested vote from peers 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021), 8845a0bcc9d74fa19808e003237576f2 (127.0.103.6:44871)
I20250111 02:53:21.124085 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:53:21.124179 4319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dc432e24e7449c879c2c35ad1559d4" candidate_uuid: "18a7cd9ed0644218857e3da957957cf8" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "8845a0bcc9d74fa19808e003237576f2"
I20250111 02:53:21.124264 3916 raft_consensus.cc:3049] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:53:21.124366 3916 raft_consensus.cc:738] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4667e917a8b44d0384ced54357111047, State: Running, Role: LEADER
I20250111 02:53:21.124393 4319 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:53:21.124639 3916 consensus_queue.cc:260] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:21.124943 3916 raft_consensus.cc:3054] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:53:21.126828 4319 raft_consensus.cc:2463] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 3.
I20250111 02:53:21.127151 3916 raft_consensus.cc:2463] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18a7cd9ed0644218857e3da957957cf8 in term 3.
I20250111 02:53:21.127202 3737 leader_election.cc:304] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18a7cd9ed0644218857e3da957957cf8, 8845a0bcc9d74fa19808e003237576f2; no voters:
I20250111 02:53:21.127467 4907 raft_consensus.cc:2798] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:53:21.127828 4907 raft_consensus.cc:695] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 3 LEADER]: Becoming Leader. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:21.128062 4907 consensus_queue.cc:237] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } }
I20250111 02:53:21.130178 3670 catalog_manager.cc:5526] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 reported cstate change: term changed from 2 to 3, leader changed from 4667e917a8b44d0384ced54357111047 (127.0.103.3) to 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1). New cstate: current_term: 3 leader_uuid: "18a7cd9ed0644218857e3da957957cf8" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:21.625133 4319 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 3 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250111 02:53:21.625648 4907 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8845a0bcc9d74fa19808e003237576f2" member_type: VOTER last_known_addr { host: "127.0.103.6" port: 44871 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:21.629890 3916 raft_consensus.cc:1270] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Refusing update from remote peer 18a7cd9ed0644218857e3da957957cf8: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250111 02:53:21.630447 4932 consensus_queue.cc:1035] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250111 02:53:22.965448 3723 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:22.967995 3770 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38"
dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
mode: GRACEFUL
new_leader_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:44968
I20250111 02:53:22.968242 3770 raft_consensus.cc:604] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Received request to transfer leadership to TS 912e2307407d45dd88991584cd0b914d
I20250111 02:53:22.968670 3723 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:22.968871 3723 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:23.119987 4937 raft_consensus.cc:988] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8: : Instructing follower 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:23.120167 4937 raft_consensus.cc:1076] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Signalling peer 912e2307407d45dd88991584cd0b914d to start an election
I20250111 02:53:23.120725 3841 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38"
dest_uuid: "912e2307407d45dd88991584cd0b914d"
from {username='slave'} at 127.0.0.1:60314
I20250111 02:53:23.120906 3841 raft_consensus.cc:491] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:23.120991 3841 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:23.123617 3841 raft_consensus.cc:513] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:23.124238 3841 leader_election.cc:290] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 5 election: Requested vote from peers 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1:42435), 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021)
I20250111 02:53:23.124634 3916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "4667e917a8b44d0384ced54357111047"
I20250111 02:53:23.124676 3770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e921916fa0e4883a3028f4fa013bc38" candidate_uuid: "912e2307407d45dd88991584cd0b914d" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "18a7cd9ed0644218857e3da957957cf8"
I20250111 02:53:23.124817 3916 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:23.124820 3770 raft_consensus.cc:3049] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Stepping down as leader of term 4
I20250111 02:53:23.124912 3770 raft_consensus.cc:738] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 18a7cd9ed0644218857e3da957957cf8, State: Running, Role: LEADER
I20250111 02:53:23.125119 3770 consensus_queue.cc:260] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:23.125365 3770 raft_consensus.cc:3054] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 4 FOLLOWER]: Advancing to term 5
I20250111 02:53:23.126816 3916 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 5.
I20250111 02:53:23.127051 3770 raft_consensus.cc:2463] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 912e2307407d45dd88991584cd0b914d in term 5.
I20250111 02:53:23.127238 3810 leader_election.cc:304] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4667e917a8b44d0384ced54357111047, 912e2307407d45dd88991584cd0b914d; no voters:
I20250111 02:53:23.127534 4893 raft_consensus.cc:2798] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Leader election won for term 5
I20250111 02:53:23.127926 4893 raft_consensus.cc:695] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Becoming Leader. State: Replica: 912e2307407d45dd88991584cd0b914d, State: Running, Role: LEADER
I20250111 02:53:23.128129 4893 consensus_queue.cc:237] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } }
I20250111 02:53:23.130239 3666 catalog_manager.cc:5526] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d reported cstate change: term changed from 4 to 5, leader changed from 18a7cd9ed0644218857e3da957957cf8 (127.0.103.1) to 912e2307407d45dd88991584cd0b914d (127.0.103.2). New cstate: current_term: 5 leader_uuid: "912e2307407d45dd88991584cd0b914d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18a7cd9ed0644218857e3da957957cf8" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 42435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "912e2307407d45dd88991584cd0b914d" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4667e917a8b44d0384ced54357111047" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 41021 } health_report { overall_health: UNKNOWN } } }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:195: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250111 02:53:23.191351 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:53:23.205387 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:23.205657 412 tablet_replica.cc:331] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:23.205835 412 raft_consensus.cc:2238] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 7 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.206167 412 raft_consensus.cc:2267] T 3bf60fc472f742518bd4328f9e88b436 P 18a7cd9ed0644218857e3da957957cf8 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.206815 412 tablet_replica.cc:331] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:23.206971 412 raft_consensus.cc:2238] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 3 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.207238 412 raft_consensus.cc:2267] T b3dc432e24e7449c879c2c35ad1559d4 P 18a7cd9ed0644218857e3da957957cf8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.207765 412 tablet_replica.cc:331] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:23.207906 412 raft_consensus.cc:2238] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.208094 412 raft_consensus.cc:2267] T 6c67b2576c5f45b3b4e291de305ec91c P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.208511 412 tablet_replica.cc:331] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:23.208657 412 raft_consensus.cc:2238] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.208866 412 raft_consensus.cc:2267] T 358018d07a3d4af0a2102356c90436c0 P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.209304 412 tablet_replica.cc:331] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:23.209434 412 raft_consensus.cc:2238] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.209664 412 raft_consensus.cc:2267] T e886897718de4e46833c86f6412bdf1a P 18a7cd9ed0644218857e3da957957cf8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.210131 412 tablet_replica.cc:331] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8: stopping tablet replica
I20250111 02:53:23.210264 412 raft_consensus.cc:2238] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.210390 412 raft_consensus.cc:2267] T 0e921916fa0e4883a3028f4fa013bc38 P 18a7cd9ed0644218857e3da957957cf8 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.226265 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:53:23.231299 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:53:23.257193 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:23.257511 412 tablet_replica.cc:331] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:23.257689 412 raft_consensus.cc:2238] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 9 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.258045 412 raft_consensus.cc:2267] T 7cf22853b8e742d98610db9dbcfea1c0 P 912e2307407d45dd88991584cd0b914d [term 9 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.258643 412 tablet_replica.cc:331] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:23.258800 412 raft_consensus.cc:2238] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.259001 412 raft_consensus.cc:2267] T e7bedf7b15ad48319668c0000adaf421 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.259451 412 tablet_replica.cc:331] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:23.259575 412 raft_consensus.cc:2238] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.259804 412 raft_consensus.cc:2267] T 1be25a13ddfe49af9a73cb6f630f505e P 912e2307407d45dd88991584cd0b914d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.260232 412 tablet_replica.cc:331] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:23.260360 412 raft_consensus.cc:2238] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 7 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.260478 412 raft_consensus.cc:2267] T 3bf60fc472f742518bd4328f9e88b436 P 912e2307407d45dd88991584cd0b914d [term 7 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.260957 412 tablet_replica.cc:331] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:23.261065 412 raft_consensus.cc:2238] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.261195 412 raft_consensus.cc:2267] T d08873d252254c8dbda59ce10b6766cb P 912e2307407d45dd88991584cd0b914d [term 4 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.261627 412 tablet_replica.cc:331] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: stopping tablet replica
I20250111 02:53:23.261760 412 raft_consensus.cc:2238] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.261979 412 pending_rounds.cc:62] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: Trying to abort 1 pending ops.
I20250111 02:53:23.262042 412 pending_rounds.cc:69] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d: Aborting op as it isn't in flight: id { term: 5 index: 31 } timestamp: 7112966156814794752 op_type: NO_OP noop_request { }
I20250111 02:53:23.262164 412 raft_consensus.cc:2883] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 5 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:53:23.262248 412 raft_consensus.cc:2267] T 0e921916fa0e4883a3028f4fa013bc38 P 912e2307407d45dd88991584cd0b914d [term 5 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.279419 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:53:23.284512 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:53:23.294051 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:23.294343 412 tablet_replica.cc:331] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:23.294518 412 raft_consensus.cc:2238] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.294692 412 raft_consensus.cc:2267] T b3dc432e24e7449c879c2c35ad1559d4 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.295250 412 tablet_replica.cc:331] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:23.295403 412 raft_consensus.cc:2238] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.295533 412 raft_consensus.cc:2267] T 1be25a13ddfe49af9a73cb6f630f505e P 4667e917a8b44d0384ced54357111047 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.295975 412 tablet_replica.cc:331] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:23.296087 412 raft_consensus.cc:2238] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.296205 412 raft_consensus.cc:2267] T 0e921916fa0e4883a3028f4fa013bc38 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.296602 412 tablet_replica.cc:331] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:23.296718 412 raft_consensus.cc:2238] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 5 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.296950 412 raft_consensus.cc:2267] T e7bedf7b15ad48319668c0000adaf421 P 4667e917a8b44d0384ced54357111047 [term 5 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.297339 412 tablet_replica.cc:331] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:23.297456 412 raft_consensus.cc:2238] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.297684 412 raft_consensus.cc:2267] T 1e52445200d144479f643c05ecb3a0f8 P 4667e917a8b44d0384ced54357111047 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.298117 412 tablet_replica.cc:331] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047: stopping tablet replica
I20250111 02:53:23.298228 412 raft_consensus.cc:2238] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.298362 412 raft_consensus.cc:2267] T d08873d252254c8dbda59ce10b6766cb P 4667e917a8b44d0384ced54357111047 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.316031 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:53:23.321230 412 tablet_server.cc:178] TabletServer@127.0.103.4:0 shutting down...
I20250111 02:53:23.345885 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:23.346130 412 tablet_replica.cc:331] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e: stopping tablet replica
I20250111 02:53:23.346331 412 raft_consensus.cc:2238] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.346539 412 raft_consensus.cc:2267] T 1e52445200d144479f643c05ecb3a0f8 P a9e5558f33db487e97343f676589083e [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.347110 412 tablet_replica.cc:331] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e: stopping tablet replica
I20250111 02:53:23.347266 412 raft_consensus.cc:2238] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.347477 412 raft_consensus.cc:2267] T e886897718de4e46833c86f6412bdf1a P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Raft consensus is shut down!
W20250111 02:53:23.347980 4282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.3:41021: connect: Connection refused (error 111)
I20250111 02:53:23.348047 412 tablet_replica.cc:331] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e: stopping tablet replica
I20250111 02:53:23.348158 412 raft_consensus.cc:2238] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.348387 412 raft_consensus.cc:2267] T 358018d07a3d4af0a2102356c90436c0 P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.348889 412 tablet_replica.cc:331] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e: stopping tablet replica
W20250111 02:53:23.348973 4282 consensus_peers.cc:487] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 -> Peer 4667e917a8b44d0384ced54357111047 (127.0.103.3:41021): Couldn't send request to peer 4667e917a8b44d0384ced54357111047. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.3:41021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:23.349059 412 raft_consensus.cc:2238] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.349323 412 raft_consensus.cc:2267] T 25a4915f163f405fa38c25cc03ceacc9 P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.349713 412 tablet_replica.cc:331] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e: stopping tablet replica
I20250111 02:53:23.349804 412 raft_consensus.cc:2238] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 9 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.349937 412 raft_consensus.cc:2267] T 7cf22853b8e742d98610db9dbcfea1c0 P a9e5558f33db487e97343f676589083e [term 9 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.350342 412 tablet_replica.cc:331] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e: stopping tablet replica
I20250111 02:53:23.350473 412 raft_consensus.cc:2238] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.350672 412 raft_consensus.cc:2267] T 6c67b2576c5f45b3b4e291de305ec91c P a9e5558f33db487e97343f676589083e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.364977 412 tablet_server.cc:195] TabletServer@127.0.103.4:0 shutdown complete.
I20250111 02:53:23.368817 412 tablet_server.cc:178] TabletServer@127.0.103.5:0 shutting down...
I20250111 02:53:23.378144 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:23.378422 412 tablet_replica.cc:331] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d: stopping tablet replica
I20250111 02:53:23.378623 412 raft_consensus.cc:2238] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.378856 412 raft_consensus.cc:2267] T 25a4915f163f405fa38c25cc03ceacc9 P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.379441 412 tablet_replica.cc:331] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d: stopping tablet replica
I20250111 02:53:23.379566 412 raft_consensus.cc:2238] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 5 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.379726 412 raft_consensus.cc:2267] T e7bedf7b15ad48319668c0000adaf421 P 5a90130f2bac4d13875c4094c826566d [term 5 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.380141 412 tablet_replica.cc:331] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d: stopping tablet replica
I20250111 02:53:23.380234 412 raft_consensus.cc:2238] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.380367 412 raft_consensus.cc:2267] T 1e52445200d144479f643c05ecb3a0f8 P 5a90130f2bac4d13875c4094c826566d [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.380718 412 tablet_replica.cc:331] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d: stopping tablet replica
I20250111 02:53:23.380815 412 raft_consensus.cc:2238] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.380972 412 raft_consensus.cc:2267] T 6c67b2576c5f45b3b4e291de305ec91c P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.381333 412 tablet_replica.cc:331] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d: stopping tablet replica
I20250111 02:53:23.381428 412 raft_consensus.cc:2238] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.381556 412 raft_consensus.cc:2267] T 1be25a13ddfe49af9a73cb6f630f505e P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.381978 412 tablet_replica.cc:331] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d: stopping tablet replica
I20250111 02:53:23.382081 412 raft_consensus.cc:2238] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.382329 412 raft_consensus.cc:2267] T e886897718de4e46833c86f6412bdf1a P 5a90130f2bac4d13875c4094c826566d [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.396626 412 tablet_server.cc:195] TabletServer@127.0.103.5:0 shutdown complete.
I20250111 02:53:23.400545 412 tablet_server.cc:178] TabletServer@127.0.103.6:0 shutting down...
I20250111 02:53:23.409446 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:23.409770 412 tablet_replica.cc:331] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2: stopping tablet replica
I20250111 02:53:23.409989 412 raft_consensus.cc:2238] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.410220 412 raft_consensus.cc:2267] T 3bf60fc472f742518bd4328f9e88b436 P 8845a0bcc9d74fa19808e003237576f2 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.410771 412 tablet_replica.cc:331] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2: stopping tablet replica
I20250111 02:53:23.410897 412 raft_consensus.cc:2238] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 9 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.411013 412 raft_consensus.cc:2267] T 7cf22853b8e742d98610db9dbcfea1c0 P 8845a0bcc9d74fa19808e003237576f2 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.411425 412 tablet_replica.cc:331] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2: stopping tablet replica
I20250111 02:53:23.411523 412 raft_consensus.cc:2238] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.411664 412 raft_consensus.cc:2267] T 25a4915f163f405fa38c25cc03ceacc9 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.412041 412 tablet_replica.cc:331] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2: stopping tablet replica
I20250111 02:53:23.412130 412 raft_consensus.cc:2238] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:23.412272 412 raft_consensus.cc:2267] T b3dc432e24e7449c879c2c35ad1559d4 P 8845a0bcc9d74fa19808e003237576f2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.412701 412 tablet_replica.cc:331] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2: stopping tablet replica
I20250111 02:53:23.412825 412 raft_consensus.cc:2238] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.413074 412 raft_consensus.cc:2267] T 358018d07a3d4af0a2102356c90436c0 P 8845a0bcc9d74fa19808e003237576f2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.413585 412 tablet_replica.cc:331] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2: stopping tablet replica
I20250111 02:53:23.413724 412 raft_consensus.cc:2238] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 4 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.414021 412 raft_consensus.cc:2267] T d08873d252254c8dbda59ce10b6766cb P 8845a0bcc9d74fa19808e003237576f2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.429482 412 tablet_server.cc:195] TabletServer@127.0.103.6:0 shutdown complete.
I20250111 02:53:23.433698 412 master.cc:537] Master@127.0.103.62:35787 shutting down...
I20250111 02:53:23.456249 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:23.456470 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:23.456566 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P edd01f073a6441f39ace89b69fab87ab: stopping tablet replica
I20250111 02:53:23.470212 412 master.cc:559] Master@127.0.103.62:35787 shutdown complete.
I20250111 02:53:23.476421 412 test_util.cc:181] -----------------------------------------------
I20250111 02:53:23.476495 412 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1736563872491422-412-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (37239 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20250111 02:53:23.478281 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:33129
I20250111 02:53:23.478703 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:23.480814 4946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:23.481966 4947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:23.482838 4949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:23.487452 412 server_base.cc:1034] running on GCE node
I20250111 02:53:23.487706 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:23.487757 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:23.487797 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564003487793 us; error 0 us; skew 500 ppm
I20250111 02:53:23.487951 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:23.493717 412 webserver.cc:458] Webserver started at http://127.0.103.62:42485/ using document root <none> and password file <none>
I20250111 02:53:23.494009 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:23.494078 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:23.494206 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:23.494838 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "d17c3e33fb5e48998e85958805c00969"
format_stamp: "Formatted at 2025-01-11 02:53:23 on dist-test-slave-4xrj"
I20250111 02:53:23.497232 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:53:23.498741 4954 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.499110 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:53:23.499188 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "d17c3e33fb5e48998e85958805c00969"
format_stamp: "Formatted at 2025-01-11 02:53:23 on dist-test-slave-4xrj"
I20250111 02:53:23.499284 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:23.513070 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:23.513548 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:23.523484 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:33129
I20250111 02:53:23.523494 5005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:33129 every 8 connection(s)
I20250111 02:53:23.524705 5006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:23.528330 5006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969: Bootstrap starting.
I20250111 02:53:23.529484 5006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:23.530902 5006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969: No bootstrap required, opened a new log
I20250111 02:53:23.531410 5006 raft_consensus.cc:357] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17c3e33fb5e48998e85958805c00969" member_type: VOTER }
I20250111 02:53:23.531527 5006 raft_consensus.cc:383] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:23.531566 5006 raft_consensus.cc:738] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d17c3e33fb5e48998e85958805c00969, State: Initialized, Role: FOLLOWER
I20250111 02:53:23.531733 5006 consensus_queue.cc:260] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17c3e33fb5e48998e85958805c00969" member_type: VOTER }
I20250111 02:53:23.531852 5006 raft_consensus.cc:397] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:53:23.531893 5006 raft_consensus.cc:491] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:53:23.531944 5006 raft_consensus.cc:3054] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:23.533417 5006 raft_consensus.cc:513] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17c3e33fb5e48998e85958805c00969" member_type: VOTER }
I20250111 02:53:23.533564 5006 leader_election.cc:304] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d17c3e33fb5e48998e85958805c00969; no voters:
I20250111 02:53:23.533877 5006 leader_election.cc:290] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:53:23.533972 5009 raft_consensus.cc:2798] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:23.534415 5009 raft_consensus.cc:695] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 1 LEADER]: Becoming Leader. State: Replica: d17c3e33fb5e48998e85958805c00969, State: Running, Role: LEADER
I20250111 02:53:23.534600 5009 consensus_queue.cc:237] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17c3e33fb5e48998e85958805c00969" member_type: VOTER }
I20250111 02:53:23.534650 5006 sys_catalog.cc:564] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:53:23.535269 5011 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d17c3e33fb5e48998e85958805c00969. Latest consensus state: current_term: 1 leader_uuid: "d17c3e33fb5e48998e85958805c00969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17c3e33fb5e48998e85958805c00969" member_type: VOTER } }
I20250111 02:53:23.535225 5010 sys_catalog.cc:455] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d17c3e33fb5e48998e85958805c00969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d17c3e33fb5e48998e85958805c00969" member_type: VOTER } }
I20250111 02:53:23.535408 5011 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [sys.catalog]: This master's current role is: LEADER
I20250111 02:53:23.535516 5010 sys_catalog.cc:458] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [sys.catalog]: This master's current role is: LEADER
I20250111 02:53:23.536545 5014 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:53:23.538638 5014 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:53:23.541023 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:53:23.541718 5014 catalog_manager.cc:1348] Generated new cluster ID: dd18d161096740848b289a0a61465326
I20250111 02:53:23.541793 5014 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:53:23.552134 5014 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:53:23.552836 5014 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:53:23.559373 5014 catalog_manager.cc:5899] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969: Generated new TSK 0
I20250111 02:53:23.559605 5014 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:53:23.574112 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:23.576702 5027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:23.577713 5028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:23.578502 5030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:23.579684 412 server_base.cc:1034] running on GCE node
I20250111 02:53:23.579911 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:23.579967 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:23.580004 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564003580001 us; error 0 us; skew 500 ppm
I20250111 02:53:23.580135 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:23.581323 412 webserver.cc:458] Webserver started at http://127.0.103.1:43391/ using document root <none> and password file <none>
I20250111 02:53:23.581558 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:23.581647 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:23.581794 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:23.582463 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "4e713206f94f4883baf769d03ca2162c"
format_stamp: "Formatted at 2025-01-11 02:53:23 on dist-test-slave-4xrj"
I20250111 02:53:23.584947 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:53:23.586876 5035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.587246 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:23.587363 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "4e713206f94f4883baf769d03ca2162c"
format_stamp: "Formatted at 2025-01-11 02:53:23 on dist-test-slave-4xrj"
I20250111 02:53:23.587483 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:23.600574 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:23.601078 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:23.601576 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:23.602487 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:23.602555 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.602627 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:23.602663 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.614621 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:36929
I20250111 02:53:23.614640 5097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:36929 every 8 connection(s)
I20250111 02:53:23.616146 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:23.619923 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:23.620191 5102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:23.621284 412 server_base.cc:1034] running on GCE node
I20250111 02:53:23.621887 5098 heartbeater.cc:346] Connected to a master server at 127.0.103.62:33129
I20250111 02:53:23.622018 5098 heartbeater.cc:463] Registering TS with master...
W20250111 02:53:23.622130 5105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:23.622411 5098 heartbeater.cc:510] Master 127.0.103.62:33129 requested a full tablet report, sending...
I20250111 02:53:23.622431 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:23.622494 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:23.622539 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564003622535 us; error 0 us; skew 500 ppm
I20250111 02:53:23.622704 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:23.623323 4971 ts_manager.cc:194] Registered new tserver with Master: 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929)
I20250111 02:53:23.624193 412 webserver.cc:458] Webserver started at http://127.0.103.2:43417/ using document root <none> and password file <none>
I20250111 02:53:23.624261 4971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53158
I20250111 02:53:23.624415 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:23.624480 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:23.624639 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:23.625262 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "45dc59c61cbc42dfb01fcbd99027a196"
format_stamp: "Formatted at 2025-01-11 02:53:23 on dist-test-slave-4xrj"
I20250111 02:53:23.627535 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:53:23.628942 5110 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.629243 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:23.629326 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "45dc59c61cbc42dfb01fcbd99027a196"
format_stamp: "Formatted at 2025-01-11 02:53:23 on dist-test-slave-4xrj"
I20250111 02:53:23.629420 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:23.643182 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:23.643748 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:23.644256 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:23.645022 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:23.645073 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.645143 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:23.645179 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.657480 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:42467
I20250111 02:53:23.657502 5172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:42467 every 8 connection(s)
I20250111 02:53:23.658914 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:23.661664 5176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:23.663105 5177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:23.663975 5179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:23.664372 412 server_base.cc:1034] running on GCE node
I20250111 02:53:23.664573 5173 heartbeater.cc:346] Connected to a master server at 127.0.103.62:33129
I20250111 02:53:23.664641 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250111 02:53:23.664697 5173 heartbeater.cc:463] Registering TS with master...
W20250111 02:53:23.664707 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:23.664769 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564003664765 us; error 0 us; skew 500 ppm
I20250111 02:53:23.664922 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:23.665062 5173 heartbeater.cc:510] Master 127.0.103.62:33129 requested a full tablet report, sending...
I20250111 02:53:23.666000 4971 ts_manager.cc:194] Registered new tserver with Master: 45dc59c61cbc42dfb01fcbd99027a196 (127.0.103.2:42467)
I20250111 02:53:23.666354 412 webserver.cc:458] Webserver started at http://127.0.103.3:40285/ using document root <none> and password file <none>
I20250111 02:53:23.666567 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:23.666652 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:23.666759 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:23.667150 4971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53160
I20250111 02:53:23.667486 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "0f36c4166db84a3ba9aa5475377e7927"
format_stamp: "Formatted at 2025-01-11 02:53:23 on dist-test-slave-4xrj"
I20250111 02:53:23.669692 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:53:23.671187 5184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.671483 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:23.671571 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "0f36c4166db84a3ba9aa5475377e7927"
format_stamp: "Formatted at 2025-01-11 02:53:23 on dist-test-slave-4xrj"
I20250111 02:53:23.671685 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:23.689926 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:23.690433 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:23.690979 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:23.691710 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:23.691767 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.691838 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:23.691876 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:23.703776 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:43745
I20250111 02:53:23.703809 5246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:43745 every 8 connection(s)
I20250111 02:53:23.709837 5247 heartbeater.cc:346] Connected to a master server at 127.0.103.62:33129
I20250111 02:53:23.709990 5247 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:23.710326 5247 heartbeater.cc:510] Master 127.0.103.62:33129 requested a full tablet report, sending...
I20250111 02:53:23.711087 4971 ts_manager.cc:194] Registered new tserver with Master: 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745)
I20250111 02:53:23.711413 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006779483s
I20250111 02:53:23.711984 4971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53174
I20250111 02:53:24.625473 5098 heartbeater.cc:502] Master 127.0.103.62:33129 was elected leader, sending a full tablet report...
I20250111 02:53:24.668262 5173 heartbeater.cc:502] Master 127.0.103.62:33129 was elected leader, sending a full tablet report...
I20250111 02:53:24.713145 5247 heartbeater.cc:502] Master 127.0.103.62:33129 was elected leader, sending a full tablet report...
I20250111 02:53:24.739116 412 test_util.cc:274] Using random seed: 1352783900
I20250111 02:53:24.750716 4971 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54174:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:53:24.751364 4971 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:53:24.771458 5212 tablet_service.cc:1467] Processing CreateTablet for tablet 3278cb6291b04b9792838f01e82bb2e3 (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250111 02:53:24.771663 5210 tablet_service.cc:1467] Processing CreateTablet for tablet 0db884613c4746c79e38a78d7c3b050f (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250111 02:53:24.771981 5212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3278cb6291b04b9792838f01e82bb2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.772347 5211 tablet_service.cc:1467] Processing CreateTablet for tablet 2da98881d597448e91a84719549d486d (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250111 02:53:24.772641 5211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da98881d597448e91a84719549d486d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.774006 5210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db884613c4746c79e38a78d7c3b050f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.776340 5267 tablet_bootstrap.cc:492] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927: Bootstrap starting.
I20250111 02:53:24.776786 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 3278cb6291b04b9792838f01e82bb2e3 (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250111 02:53:24.777145 5138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3278cb6291b04b9792838f01e82bb2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.778501 5267 tablet_bootstrap.cc:654] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.778708 5137 tablet_service.cc:1467] Processing CreateTablet for tablet 2da98881d597448e91a84719549d486d (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250111 02:53:24.778767 5136 tablet_service.cc:1467] Processing CreateTablet for tablet 0db884613c4746c79e38a78d7c3b050f (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250111 02:53:24.779209 5137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da98881d597448e91a84719549d486d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.779430 5136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db884613c4746c79e38a78d7c3b050f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.786502 5269 tablet_bootstrap.cc:492] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196: Bootstrap starting.
I20250111 02:53:24.787611 5063 tablet_service.cc:1467] Processing CreateTablet for tablet 3278cb6291b04b9792838f01e82bb2e3 (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250111 02:53:24.787851 5267 tablet_bootstrap.cc:492] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927: No bootstrap required, opened a new log
I20250111 02:53:24.787760 5062 tablet_service.cc:1467] Processing CreateTablet for tablet 2da98881d597448e91a84719549d486d (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250111 02:53:24.787962 5063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3278cb6291b04b9792838f01e82bb2e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.788134 5062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da98881d597448e91a84719549d486d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.788241 5269 tablet_bootstrap.cc:654] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.788203 5061 tablet_service.cc:1467] Processing CreateTablet for tablet 0db884613c4746c79e38a78d7c3b050f (DEFAULT_TABLE table=test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250111 02:53:24.788636 5061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db884613c4746c79e38a78d7c3b050f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:24.794036 5271 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c: Bootstrap starting.
I20250111 02:53:24.794318 5269 tablet_bootstrap.cc:492] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196: No bootstrap required, opened a new log
I20250111 02:53:24.794584 5267 ts_tablet_manager.cc:1397] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.007s
I20250111 02:53:24.794664 5269 ts_tablet_manager.cc:1397] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20250111 02:53:24.795462 5269 raft_consensus.cc:357] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.795460 5267 raft_consensus.cc:357] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.795653 5269 raft_consensus.cc:383] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.795763 5271 tablet_bootstrap.cc:654] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.795774 5267 raft_consensus.cc:383] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.795826 5269 raft_consensus.cc:738] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dc59c61cbc42dfb01fcbd99027a196, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.795872 5267 raft_consensus.cc:738] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f36c4166db84a3ba9aa5475377e7927, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.796092 5269 consensus_queue.cc:260] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.796097 5267 consensus_queue.cc:260] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.796795 5269 ts_tablet_manager.cc:1428] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250111 02:53:24.797003 5269 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196: Bootstrap starting.
I20250111 02:53:24.798501 5269 tablet_bootstrap.cc:654] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.800367 5269 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196: No bootstrap required, opened a new log
I20250111 02:53:24.800527 5271 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c: No bootstrap required, opened a new log
I20250111 02:53:24.800551 5269 ts_tablet_manager.cc:1397] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250111 02:53:24.801033 5267 ts_tablet_manager.cc:1428] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20250111 02:53:24.801263 5267 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: Bootstrap starting.
I20250111 02:53:24.801381 5271 ts_tablet_manager.cc:1397] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250111 02:53:24.801407 5269 raft_consensus.cc:357] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.801615 5269 raft_consensus.cc:383] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.801708 5269 raft_consensus.cc:738] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dc59c61cbc42dfb01fcbd99027a196, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.801982 5269 consensus_queue.cc:260] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.802297 5271 raft_consensus.cc:357] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.802532 5271 raft_consensus.cc:383] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.802666 5271 raft_consensus.cc:738] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e713206f94f4883baf769d03ca2162c, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.802448 5269 ts_tablet_manager.cc:1428] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250111 02:53:24.802942 5271 consensus_queue.cc:260] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.803589 5267 tablet_bootstrap.cc:654] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.804275 5269 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196: Bootstrap starting.
I20250111 02:53:24.806371 5269 tablet_bootstrap.cc:654] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.809886 5271 ts_tablet_manager.cc:1428] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250111 02:53:24.810205 5271 tablet_bootstrap.cc:492] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c: Bootstrap starting.
I20250111 02:53:24.811920 5271 tablet_bootstrap.cc:654] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.813048 5272 raft_consensus.cc:491] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:53:24.813189 5272 raft_consensus.cc:513] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.813874 5267 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: No bootstrap required, opened a new log
I20250111 02:53:24.813987 5267 ts_tablet_manager.cc:1397] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20250111 02:53:24.814733 5267 raft_consensus.cc:357] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.814936 5267 raft_consensus.cc:383] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.814993 5267 raft_consensus.cc:738] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f36c4166db84a3ba9aa5475377e7927, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.815208 5272 leader_election.cc:290] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929), 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745)
I20250111 02:53:24.817286 5269 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196: No bootstrap required, opened a new log
I20250111 02:53:24.817519 5269 ts_tablet_manager.cc:1397] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20250111 02:53:24.818418 5271 tablet_bootstrap.cc:492] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c: No bootstrap required, opened a new log
I20250111 02:53:24.818327 5269 raft_consensus.cc:357] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.818586 5271 ts_tablet_manager.cc:1397] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20250111 02:53:24.818616 5269 raft_consensus.cc:383] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.818792 5269 raft_consensus.cc:738] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45dc59c61cbc42dfb01fcbd99027a196, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.819047 5269 consensus_queue.cc:260] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.819604 5269 ts_tablet_manager.cc:1428] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250111 02:53:24.819823 5271 raft_consensus.cc:357] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.821036 5271 raft_consensus.cc:383] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.821146 5271 raft_consensus.cc:738] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e713206f94f4883baf769d03ca2162c, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.821403 5271 consensus_queue.cc:260] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.822059 5271 ts_tablet_manager.cc:1428] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250111 02:53:24.822331 5271 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: Bootstrap starting.
I20250111 02:53:24.815214 5267 consensus_queue.cc:260] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.823176 5073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db884613c4746c79e38a78d7c3b050f" candidate_uuid: "45dc59c61cbc42dfb01fcbd99027a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e713206f94f4883baf769d03ca2162c" is_pre_election: true
I20250111 02:53:24.823973 5267 ts_tablet_manager.cc:1428] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250111 02:53:24.824081 5271 tablet_bootstrap.cc:654] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.824221 5267 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927: Bootstrap starting.
W20250111 02:53:24.825824 5114 leader_election.cc:343] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:53:24.826097 5267 tablet_bootstrap.cc:654] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:24.827111 5271 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: No bootstrap required, opened a new log
I20250111 02:53:24.827368 5271 ts_tablet_manager.cc:1397] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250111 02:53:24.828084 5271 raft_consensus.cc:357] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.828284 5271 raft_consensus.cc:383] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.828389 5271 raft_consensus.cc:738] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e713206f94f4883baf769d03ca2162c, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.828588 5271 consensus_queue.cc:260] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.828879 5267 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927: No bootstrap required, opened a new log
I20250111 02:53:24.829003 5267 ts_tablet_manager.cc:1397] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250111 02:53:24.829665 5267 raft_consensus.cc:357] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.829872 5267 raft_consensus.cc:383] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:24.829942 5267 raft_consensus.cc:738] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f36c4166db84a3ba9aa5475377e7927, State: Initialized, Role: FOLLOWER
I20250111 02:53:24.830200 5222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db884613c4746c79e38a78d7c3b050f" candidate_uuid: "45dc59c61cbc42dfb01fcbd99027a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f36c4166db84a3ba9aa5475377e7927" is_pre_election: true
I20250111 02:53:24.830179 5267 consensus_queue.cc:260] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.830439 5222 raft_consensus.cc:2463] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45dc59c61cbc42dfb01fcbd99027a196 in term 0.
I20250111 02:53:24.830812 5267 ts_tablet_manager.cc:1428] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:24.831195 5112 leader_election.cc:304] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0f36c4166db84a3ba9aa5475377e7927, 45dc59c61cbc42dfb01fcbd99027a196; no voters: 4e713206f94f4883baf769d03ca2162c
I20250111 02:53:24.831441 5272 raft_consensus.cc:2798] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:53:24.831513 5272 raft_consensus.cc:491] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:53:24.831575 5272 raft_consensus.cc:3054] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.833148 5272 raft_consensus.cc:513] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.833607 5272 leader_election.cc:290] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 election: Requested vote from peers 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929), 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745)
I20250111 02:53:24.833936 5073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db884613c4746c79e38a78d7c3b050f" candidate_uuid: "45dc59c61cbc42dfb01fcbd99027a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e713206f94f4883baf769d03ca2162c"
I20250111 02:53:24.834273 5222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db884613c4746c79e38a78d7c3b050f" candidate_uuid: "45dc59c61cbc42dfb01fcbd99027a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f36c4166db84a3ba9aa5475377e7927"
I20250111 02:53:24.834398 5222 raft_consensus.cc:3054] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.836079 5222 raft_consensus.cc:2463] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45dc59c61cbc42dfb01fcbd99027a196 in term 1.
I20250111 02:53:24.836374 5112 leader_election.cc:304] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f36c4166db84a3ba9aa5475377e7927, 45dc59c61cbc42dfb01fcbd99027a196; no voters:
I20250111 02:53:24.836596 5272 raft_consensus.cc:2798] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:24.837004 5272 raft_consensus.cc:695] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 LEADER]: Becoming Leader. State: Replica: 45dc59c61cbc42dfb01fcbd99027a196, State: Running, Role: LEADER
I20250111 02:53:24.837368 5276 raft_consensus.cc:491] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:53:24.837309 5272 consensus_queue.cc:237] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.837481 5276 raft_consensus.cc:513] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.838191 5276 leader_election.cc:290] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45dc59c61cbc42dfb01fcbd99027a196 (127.0.103.2:42467), 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745)
I20250111 02:53:24.842382 5284 raft_consensus.cc:491] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:53:24.842566 5284 raft_consensus.cc:513] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.843391 5222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3278cb6291b04b9792838f01e82bb2e3" candidate_uuid: "45dc59c61cbc42dfb01fcbd99027a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f36c4166db84a3ba9aa5475377e7927" is_pre_election: true
I20250111 02:53:24.843550 5222 raft_consensus.cc:2463] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45dc59c61cbc42dfb01fcbd99027a196 in term 0.
I20250111 02:53:24.843889 5112 leader_election.cc:304] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f36c4166db84a3ba9aa5475377e7927, 45dc59c61cbc42dfb01fcbd99027a196; no voters:
I20250111 02:53:24.847152 5073 raft_consensus.cc:3054] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.847404 5271 ts_tablet_manager.cc:1428] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: Time spent starting tablet: real 0.020s user 0.018s sys 0.001s
I20250111 02:53:24.847843 5284 leader_election.cc:290] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745), 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929)
I20250111 02:53:24.847990 5284 raft_consensus.cc:2798] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:53:24.848073 5284 raft_consensus.cc:491] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:53:24.848150 5284 raft_consensus.cc:3054] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.848229 5072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3278cb6291b04b9792838f01e82bb2e3" candidate_uuid: "45dc59c61cbc42dfb01fcbd99027a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e713206f94f4883baf769d03ca2162c" is_pre_election: true
I20250111 02:53:24.848395 5072 raft_consensus.cc:2463] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45dc59c61cbc42dfb01fcbd99027a196 in term 0.
I20250111 02:53:24.850144 5073 raft_consensus.cc:2463] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45dc59c61cbc42dfb01fcbd99027a196 in term 1.
I20250111 02:53:24.851140 5284 raft_consensus.cc:513] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.851738 5284 leader_election.cc:290] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 election: Requested vote from peers 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745), 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929)
I20250111 02:53:24.852253 5073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3278cb6291b04b9792838f01e82bb2e3" candidate_uuid: "45dc59c61cbc42dfb01fcbd99027a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e713206f94f4883baf769d03ca2162c"
I20250111 02:53:24.852463 5073 raft_consensus.cc:3054] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.852732 5222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3278cb6291b04b9792838f01e82bb2e3" candidate_uuid: "45dc59c61cbc42dfb01fcbd99027a196" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f36c4166db84a3ba9aa5475377e7927"
I20250111 02:53:24.852872 5222 raft_consensus.cc:3054] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.855037 5222 raft_consensus.cc:2463] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45dc59c61cbc42dfb01fcbd99027a196 in term 1.
I20250111 02:53:24.855412 5112 leader_election.cc:304] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f36c4166db84a3ba9aa5475377e7927, 45dc59c61cbc42dfb01fcbd99027a196; no voters:
I20250111 02:53:24.855787 5073 raft_consensus.cc:2463] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45dc59c61cbc42dfb01fcbd99027a196 in term 1.
I20250111 02:53:24.855798 5288 raft_consensus.cc:2798] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:24.856134 5288 raft_consensus.cc:695] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 LEADER]: Becoming Leader. State: Replica: 45dc59c61cbc42dfb01fcbd99027a196, State: Running, Role: LEADER
I20250111 02:53:24.856577 5288 consensus_queue.cc:237] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } }
I20250111 02:53:24.863852 4969 catalog_manager.cc:5526] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45dc59c61cbc42dfb01fcbd99027a196 (127.0.103.2). New cstate: current_term: 1 leader_uuid: "45dc59c61cbc42dfb01fcbd99027a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:24.866663 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da98881d597448e91a84719549d486d" candidate_uuid: "4e713206f94f4883baf769d03ca2162c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dc59c61cbc42dfb01fcbd99027a196" is_pre_election: true
I20250111 02:53:24.866880 5148 raft_consensus.cc:2463] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e713206f94f4883baf769d03ca2162c in term 0.
W20250111 02:53:24.867131 5099 tablet.cc:2367] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:53:24.867165 5222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da98881d597448e91a84719549d486d" candidate_uuid: "4e713206f94f4883baf769d03ca2162c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f36c4166db84a3ba9aa5475377e7927" is_pre_election: true
I20250111 02:53:24.867308 5222 raft_consensus.cc:2463] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e713206f94f4883baf769d03ca2162c in term 0.
I20250111 02:53:24.867491 5038 leader_election.cc:304] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dc59c61cbc42dfb01fcbd99027a196, 4e713206f94f4883baf769d03ca2162c; no voters:
I20250111 02:53:24.867717 5276 raft_consensus.cc:2798] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:53:24.867823 5276 raft_consensus.cc:491] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:53:24.867914 5276 raft_consensus.cc:3054] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.868306 4969 catalog_manager.cc:5526] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45dc59c61cbc42dfb01fcbd99027a196 (127.0.103.2). New cstate: current_term: 1 leader_uuid: "45dc59c61cbc42dfb01fcbd99027a196" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:24.869896 5276 raft_consensus.cc:513] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.870391 5276 leader_election.cc:290] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [CANDIDATE]: Term 1 election: Requested vote from peers 45dc59c61cbc42dfb01fcbd99027a196 (127.0.103.2:42467), 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745)
I20250111 02:53:24.870675 5148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da98881d597448e91a84719549d486d" candidate_uuid: "4e713206f94f4883baf769d03ca2162c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45dc59c61cbc42dfb01fcbd99027a196"
I20250111 02:53:24.870828 5148 raft_consensus.cc:3054] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.870777 5222 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2da98881d597448e91a84719549d486d" candidate_uuid: "4e713206f94f4883baf769d03ca2162c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f36c4166db84a3ba9aa5475377e7927"
I20250111 02:53:24.870935 5222 raft_consensus.cc:3054] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:24.872442 5148 raft_consensus.cc:2463] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e713206f94f4883baf769d03ca2162c in term 1.
I20250111 02:53:24.872478 5222 raft_consensus.cc:2463] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e713206f94f4883baf769d03ca2162c in term 1.
I20250111 02:53:24.872749 5038 leader_election.cc:304] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45dc59c61cbc42dfb01fcbd99027a196, 4e713206f94f4883baf769d03ca2162c; no voters:
I20250111 02:53:24.872954 5276 raft_consensus.cc:2798] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:24.873292 5276 raft_consensus.cc:695] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 1 LEADER]: Becoming Leader. State: Replica: 4e713206f94f4883baf769d03ca2162c, State: Running, Role: LEADER
I20250111 02:53:24.873512 5276 consensus_queue.cc:237] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } }
I20250111 02:53:24.875434 4969 catalog_manager.cc:5526] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e713206f94f4883baf769d03ca2162c (127.0.103.1). New cstate: current_term: 1 leader_uuid: "4e713206f94f4883baf769d03ca2162c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:24.886209 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:24.888796 5292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:24.889417 5293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:24.890399 5295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:24.891258 412 server_base.cc:1034] running on GCE node
I20250111 02:53:24.891469 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:24.891512 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:24.891533 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564004891531 us; error 0 us; skew 500 ppm
I20250111 02:53:24.891654 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:24.892915 412 webserver.cc:458] Webserver started at http://127.0.103.4:38759/ using document root <none> and password file <none>
I20250111 02:53:24.893100 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:24.893155 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:24.893239 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:24.893785 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-3-root/instance:
uuid: "a99b5d3fae6040afa0fd48c6340ce0bd"
format_stamp: "Formatted at 2025-01-11 02:53:24 on dist-test-slave-4xrj"
I20250111 02:53:24.896109 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:24.897579 5300 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:24.897922 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:24.898021 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-3-root
uuid: "a99b5d3fae6040afa0fd48c6340ce0bd"
format_stamp: "Formatted at 2025-01-11 02:53:24 on dist-test-slave-4xrj"
I20250111 02:53:24.898121 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:24.906044 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:24.906477 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:24.906957 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:24.907698 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:24.907757 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:24.907836 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:24.907882 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:24.919306 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.4:42643
I20250111 02:53:24.919327 5362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.4:42643 every 8 connection(s)
I20250111 02:53:24.925643 5363 heartbeater.cc:346] Connected to a master server at 127.0.103.62:33129
I20250111 02:53:24.925776 5363 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:24.926091 5363 heartbeater.cc:510] Master 127.0.103.62:33129 requested a full tablet report, sending...
I20250111 02:53:24.926780 4969 ts_manager.cc:194] Registered new tserver with Master: a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4:42643)
I20250111 02:53:24.927454 4969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54182
I20250111 02:53:24.931106 5272 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:53:24.933355 5288 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250111 02:53:24.933416 5288 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250111 02:53:24.939610 5272 consensus_queue.cc:1035] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:53:24.941610 5288 consensus_queue.cc:1035] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:53:24.942065 5288 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:53:24.957544 5276 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:53:24.963749 5375 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:53:25.540905 5024 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:25.547446 5147 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0db884613c4746c79e38a78d7c3b050f"
dest_uuid: "45dc59c61cbc42dfb01fcbd99027a196"
mode: GRACEFUL
new_leader_uuid: "0f36c4166db84a3ba9aa5475377e7927"
from {username='slave'} at 127.0.0.1:43974
I20250111 02:53:25.547776 5148 consensus_queue.cc:237] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.548736 5147 raft_consensus.cc:604] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 LEADER]: Received request to transfer leadership to TS 0f36c4166db84a3ba9aa5475377e7927
I20250111 02:53:25.549611 5222 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Refusing update from remote peer 45dc59c61cbc42dfb01fcbd99027a196: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:53:25.549791 5024 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250111 02:53:25.549939 5024 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:25.550305 5373 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:53:25.550715 5072 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Refusing update from remote peer 45dc59c61cbc42dfb01fcbd99027a196: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:53:25.551220 5373 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:53:25.552884 5222 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } } }
I20250111 02:53:25.553751 5072 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } } }
W20250111 02:53:25.557276 5114 consensus_peers.cc:487] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 -> Peer a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4:42643): Couldn't send request to peer a99b5d3fae6040afa0fd48c6340ce0bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0db884613c4746c79e38a78d7c3b050f. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:25.557740 4969 catalog_manager.cc:5526] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c reported cstate change: config changed from index -1 to 2, NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "45dc59c61cbc42dfb01fcbd99027a196" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } } }
I20250111 02:53:25.591351 5288 raft_consensus.cc:988] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196: : Instructing follower 0f36c4166db84a3ba9aa5475377e7927 to start an election
I20250111 02:53:25.591558 5288 raft_consensus.cc:1076] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 LEADER]: Signalling peer 0f36c4166db84a3ba9aa5475377e7927 to start an election
I20250111 02:53:25.592226 5222 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0db884613c4746c79e38a78d7c3b050f"
dest_uuid: "0f36c4166db84a3ba9aa5475377e7927"
from {username='slave'} at 127.0.0.1:49168
I20250111 02:53:25.592413 5222 raft_consensus.cc:491] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:53:25.592496 5222 raft_consensus.cc:3054] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:25.594156 5222 raft_consensus.cc:513] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.594789 5222 leader_election.cc:290] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [CANDIDATE]: Term 2 election: Requested vote from peers 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929), 45dc59c61cbc42dfb01fcbd99027a196 (127.0.103.2:42467)
I20250111 02:53:25.601442 5072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db884613c4746c79e38a78d7c3b050f" candidate_uuid: "0f36c4166db84a3ba9aa5475377e7927" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "4e713206f94f4883baf769d03ca2162c"
I20250111 02:53:25.601616 5072 raft_consensus.cc:3054] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:25.603327 5072 raft_consensus.cc:2463] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f36c4166db84a3ba9aa5475377e7927 in term 2.
I20250111 02:53:25.603714 5188 leader_election.cc:304] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f36c4166db84a3ba9aa5475377e7927, 4e713206f94f4883baf769d03ca2162c; no voters:
I20250111 02:53:25.603919 5147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0db884613c4746c79e38a78d7c3b050f" candidate_uuid: "0f36c4166db84a3ba9aa5475377e7927" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "45dc59c61cbc42dfb01fcbd99027a196"
I20250111 02:53:25.604039 5383 raft_consensus.cc:2798] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:53:25.604102 5147 raft_consensus.cc:3049] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 LEADER]: Stepping down as leader of term 1
I20250111 02:53:25.604163 5147 raft_consensus.cc:738] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 45dc59c61cbc42dfb01fcbd99027a196, State: Running, Role: LEADER
I20250111 02:53:25.604435 5383 raft_consensus.cc:695] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 2 LEADER]: Becoming Leader. State: Replica: 0f36c4166db84a3ba9aa5475377e7927, State: Running, Role: LEADER
I20250111 02:53:25.604449 5147 consensus_queue.cc:260] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.604736 5383 consensus_queue.cc:237] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.604930 5147 raft_consensus.cc:3054] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:53:25.606844 5147 raft_consensus.cc:2463] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f36c4166db84a3ba9aa5475377e7927 in term 2.
I20250111 02:53:25.662267 4971 catalog_manager.cc:5526] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 reported cstate change: term changed from 1 to 2, leader changed from 45dc59c61cbc42dfb01fcbd99027a196 (127.0.103.2) to 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3). New cstate: current_term: 2 leader_uuid: "0f36c4166db84a3ba9aa5475377e7927" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:25.686633 5147 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Refusing update from remote peer 0f36c4166db84a3ba9aa5475377e7927: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:53:25.687162 5383 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:53:25.688050 5147 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } } }
I20250111 02:53:25.696627 5072 consensus_queue.cc:237] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.698474 5147 raft_consensus.cc:1270] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Refusing update from remote peer 4e713206f94f4883baf769d03ca2162c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:53:25.698809 5222 raft_consensus.cc:1270] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Refusing update from remote peer 4e713206f94f4883baf769d03ca2162c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:53:25.698977 5392 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:53:25.699254 5375 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:53:25.701668 5147 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } } }
I20250111 02:53:25.702087 5222 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } } }
W20250111 02:53:25.704304 5039 consensus_peers.cc:487] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c -> Peer a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4:42643): Couldn't send request to peer a99b5d3fae6040afa0fd48c6340ce0bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2da98881d597448e91a84719549d486d. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:25.705969 5072 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 2 FOLLOWER]: Refusing update from remote peer 0f36c4166db84a3ba9aa5475377e7927: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:53:25.706377 5389 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250111 02:53:25.711395 5188 consensus_peers.cc:487] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 -> Peer a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4:42643): Couldn't send request to peer a99b5d3fae6040afa0fd48c6340ce0bd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0db884613c4746c79e38a78d7c3b050f. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:25.714197 5392 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } } }
I20250111 02:53:25.726763 4970 catalog_manager.cc:5526] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 reported cstate change: config changed from index -1 to 2, NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "4e713206f94f4883baf769d03ca2162c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } } }
I20250111 02:53:25.788486 5399 ts_tablet_manager.cc:927] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: Initiating tablet copy from peer 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745)
I20250111 02:53:25.789233 5399 tablet_copy_client.cc:323] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:43745
I20250111 02:53:25.794641 5232 tablet_copy_service.cc:140] P 0f36c4166db84a3ba9aa5475377e7927: Received BeginTabletCopySession request for tablet 0db884613c4746c79e38a78d7c3b050f from peer a99b5d3fae6040afa0fd48c6340ce0bd ({username='slave'} at 127.0.0.1:49174)
I20250111 02:53:25.794773 5232 tablet_copy_service.cc:161] P 0f36c4166db84a3ba9aa5475377e7927: Beginning new tablet copy session on tablet 0db884613c4746c79e38a78d7c3b050f from peer a99b5d3fae6040afa0fd48c6340ce0bd at {username='slave'} at 127.0.0.1:49174: session id = a99b5d3fae6040afa0fd48c6340ce0bd-0db884613c4746c79e38a78d7c3b050f
I20250111 02:53:25.796550 5232 tablet_copy_source_session.cc:215] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:25.797516 5399 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0db884613c4746c79e38a78d7c3b050f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:25.801592 5399 tablet_copy_client.cc:806] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:25.801820 5399 tablet_copy_client.cc:670] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:25.803144 5399 tablet_copy_client.cc:538] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:25.805357 5399 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: Bootstrap starting.
I20250111 02:53:25.809881 5404 ts_tablet_manager.cc:927] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: Initiating tablet copy from peer 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929)
I20250111 02:53:25.811342 5404 tablet_copy_client.cc:323] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.1:36929
I20250111 02:53:25.811594 5399 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:25.812006 5399 tablet_bootstrap.cc:492] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: Bootstrap complete.
I20250111 02:53:25.812327 5399 ts_tablet_manager.cc:1397] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: Time spent bootstrapping tablet: real 0.007s user 0.009s sys 0.000s
I20250111 02:53:25.813063 5399 raft_consensus.cc:357] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.813302 5399 raft_consensus.cc:738] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: a99b5d3fae6040afa0fd48c6340ce0bd, State: Initialized, Role: LEARNER
I20250111 02:53:25.813503 5399 consensus_queue.cc:260] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.814319 5363 heartbeater.cc:502] Master 127.0.103.62:33129 was elected leader, sending a full tablet report...
I20250111 02:53:25.814356 5399 ts_tablet_manager.cc:1428] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250111 02:53:25.814944 5232 tablet_copy_service.cc:342] P 0f36c4166db84a3ba9aa5475377e7927: Request end of tablet copy session a99b5d3fae6040afa0fd48c6340ce0bd-0db884613c4746c79e38a78d7c3b050f received from {username='slave'} at 127.0.0.1:49174
I20250111 02:53:25.815052 5232 tablet_copy_service.cc:434] P 0f36c4166db84a3ba9aa5475377e7927: ending tablet copy session a99b5d3fae6040afa0fd48c6340ce0bd-0db884613c4746c79e38a78d7c3b050f on tablet 0db884613c4746c79e38a78d7c3b050f with peer a99b5d3fae6040afa0fd48c6340ce0bd
I20250111 02:53:25.818405 5083 tablet_copy_service.cc:140] P 4e713206f94f4883baf769d03ca2162c: Received BeginTabletCopySession request for tablet 2da98881d597448e91a84719549d486d from peer a99b5d3fae6040afa0fd48c6340ce0bd ({username='slave'} at 127.0.0.1:60118)
I20250111 02:53:25.818558 5083 tablet_copy_service.cc:161] P 4e713206f94f4883baf769d03ca2162c: Beginning new tablet copy session on tablet 2da98881d597448e91a84719549d486d from peer a99b5d3fae6040afa0fd48c6340ce0bd at {username='slave'} at 127.0.0.1:60118: session id = a99b5d3fae6040afa0fd48c6340ce0bd-2da98881d597448e91a84719549d486d
I20250111 02:53:25.820703 5083 tablet_copy_source_session.cc:215] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:25.821957 5404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2da98881d597448e91a84719549d486d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:25.826268 5404 tablet_copy_client.cc:806] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:25.826479 5404 tablet_copy_client.cc:670] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:25.827788 5404 tablet_copy_client.cc:538] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:25.830236 5404 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: Bootstrap starting.
I20250111 02:53:25.835714 5404 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:25.836139 5404 tablet_bootstrap.cc:492] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: Bootstrap complete.
I20250111 02:53:25.836388 5404 ts_tablet_manager.cc:1397] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20250111 02:53:25.837013 5404 raft_consensus.cc:357] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.837245 5404 raft_consensus.cc:738] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a99b5d3fae6040afa0fd48c6340ce0bd, State: Initialized, Role: LEARNER
I20250111 02:53:25.837411 5404 consensus_queue.cc:260] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: true } }
I20250111 02:53:25.837925 5404 ts_tablet_manager.cc:1428] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250111 02:53:25.838513 5083 tablet_copy_service.cc:342] P 4e713206f94f4883baf769d03ca2162c: Request end of tablet copy session a99b5d3fae6040afa0fd48c6340ce0bd-2da98881d597448e91a84719549d486d received from {username='slave'} at 127.0.0.1:60118
I20250111 02:53:25.838644 5083 tablet_copy_service.cc:434] P 4e713206f94f4883baf769d03ca2162c: ending tablet copy session a99b5d3fae6040afa0fd48c6340ce0bd-2da98881d597448e91a84719549d486d on tablet 2da98881d597448e91a84719549d486d with peer a99b5d3fae6040afa0fd48c6340ce0bd
I20250111 02:53:25.887529 5338 raft_consensus.cc:1212] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 LEARNER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20250111 02:53:25.896984 5338 raft_consensus.cc:1212] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250111 02:53:26.030261 5383 raft_consensus.cc:1059] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927: attempting to promote NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd to VOTER
I20250111 02:53:26.030987 5383 consensus_queue.cc:237] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } }
I20250111 02:53:26.032718 5338 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 LEARNER]: Refusing update from remote peer 0f36c4166db84a3ba9aa5475377e7927: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250111 02:53:26.032720 5147 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Refusing update from remote peer 0f36c4166db84a3ba9aa5475377e7927: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250111 02:53:26.032940 5073 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 2 FOLLOWER]: Refusing update from remote peer 0f36c4166db84a3ba9aa5475377e7927: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250111 02:53:26.033250 5398 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:26.033520 5389 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:26.034149 5383 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:26.036504 5073 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.037272 5338 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.040647 4971 catalog_manager.cc:5526] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c reported cstate change: config changed from index 2 to 4, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0f36c4166db84a3ba9aa5475377e7927" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.062908 5375 raft_consensus.cc:1059] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c: attempting to promote NON_VOTER a99b5d3fae6040afa0fd48c6340ce0bd to VOTER
I20250111 02:53:26.063719 5375 consensus_queue.cc:237] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } }
I20250111 02:53:26.065639 5338 raft_consensus.cc:1270] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 LEARNER]: Refusing update from remote peer 4e713206f94f4883baf769d03ca2162c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:26.065925 5147 raft_consensus.cc:1270] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Refusing update from remote peer 4e713206f94f4883baf769d03ca2162c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:26.066473 5392 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:53:26.066676 5221 raft_consensus.cc:1270] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Refusing update from remote peer 4e713206f94f4883baf769d03ca2162c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:26.067405 5392 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:53:26.067677 5396 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:53:26.070001 5221 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.072297 5338 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.072551 5396 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.074746 4969 catalog_manager.cc:5526] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 reported cstate change: config changed from index 2 to 3, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e713206f94f4883baf769d03ca2162c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.077167 5389 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.077634 5147 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } attrs { replace: true } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.080175 5148 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, a99b5d3fae6040afa0fd48c6340ce0bd (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } attrs { replace: true } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.193220 5072 consensus_queue.cc:237] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } }
I20250111 02:53:26.195125 5148 raft_consensus.cc:1270] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Refusing update from remote peer 4e713206f94f4883baf769d03ca2162c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:26.195109 5338 raft_consensus.cc:1270] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 FOLLOWER]: Refusing update from remote peer 4e713206f94f4883baf769d03ca2162c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:26.195062 5222 consensus_queue.cc:237] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } }
I20250111 02:53:26.195740 5375 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20250111 02:53:26.203923 5148 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Refusing update from remote peer 0f36c4166db84a3ba9aa5475377e7927: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250111 02:53:26.204082 5409 consensus_queue.cc:1035] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:26.204979 5398 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:53:26.205466 5338 raft_consensus.cc:1270] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 FOLLOWER]: Refusing update from remote peer 0f36c4166db84a3ba9aa5475377e7927: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250111 02:53:26.206199 5398 consensus_queue.cc:1035] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:53:26.208168 5147 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4e713206f94f4883baf769d03ca2162c (127.0.103.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.208364 5338 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.208169 5148 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.213316 4970 catalog_manager.cc:5526] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 reported cstate change: config changed from index 4 to 5, VOTER 4e713206f94f4883baf769d03ca2162c (127.0.103.1) evicted. New cstate: current_term: 2 leader_uuid: "0f36c4166db84a3ba9aa5475377e7927" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.214406 5337 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4e713206f94f4883baf769d03ca2162c (127.0.103.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.217562 4971 catalog_manager.cc:5526] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd reported cstate change: config changed from index 3 to 4, VOTER 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3) evicted. New cstate: current_term: 1 leader_uuid: "4e713206f94f4883baf769d03ca2162c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.228835 5061 tablet_service.cc:1514] Processing DeleteTablet for tablet 0db884613c4746c79e38a78d7c3b050f with delete_type TABLET_DATA_TOMBSTONED (TS 4e713206f94f4883baf769d03ca2162c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:60074
I20250111 02:53:26.229530 5411 tablet_replica.cc:331] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: stopping tablet replica
I20250111 02:53:26.229779 5411 raft_consensus.cc:2238] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:26.230106 5411 raft_consensus.cc:2267] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:26.230975 5411 ts_tablet_manager.cc:1905] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:26.234899 5411 ts_tablet_manager.cc:1918] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250111 02:53:26.235003 5411 log.cc:1198] T 0db884613c4746c79e38a78d7c3b050f P 4e713206f94f4883baf769d03ca2162c: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-0-root/wals/0db884613c4746c79e38a78d7c3b050f
I20250111 02:53:26.235628 4958 catalog_manager.cc:4872] TS 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929): tablet 0db884613c4746c79e38a78d7c3b050f (table test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]) successfully deleted
I20250111 02:53:26.273754 5375 raft_consensus.cc:2949] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e713206f94f4883baf769d03ca2162c" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 36929 } } peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.276393 4958 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 2da98881d597448e91a84719549d486d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:26.297780 5389 raft_consensus.cc:2949] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 4e713206f94f4883baf769d03ca2162c (127.0.103.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "45dc59c61cbc42dfb01fcbd99027a196" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 42467 } } peers { permanent_uuid: "0f36c4166db84a3ba9aa5475377e7927" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 43745 } } peers { permanent_uuid: "a99b5d3fae6040afa0fd48c6340ce0bd" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 42643 } attrs { promote: false } } }
I20250111 02:53:26.298389 5210 tablet_service.cc:1514] Processing DeleteTablet for tablet 2da98881d597448e91a84719549d486d with delete_type TABLET_DATA_TOMBSTONED (TS 0f36c4166db84a3ba9aa5475377e7927 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49160
I20250111 02:53:26.299978 5412 tablet_replica.cc:331] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: stopping tablet replica
I20250111 02:53:26.300189 5412 raft_consensus.cc:2238] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:26.300465 5412 raft_consensus.cc:2267] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:26.301091 4956 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 0db884613c4746c79e38a78d7c3b050f with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:26.301542 5412 ts_tablet_manager.cc:1905] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:53:26.307019 5412 ts_tablet_manager.cc:1918] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250111 02:53:26.307142 5412 log.cc:1198] T 2da98881d597448e91a84719549d486d P 0f36c4166db84a3ba9aa5475377e7927: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1736563872491422-412-0/minicluster-data/ts-2-root/wals/2da98881d597448e91a84719549d486d
I20250111 02:53:26.307740 4956 catalog_manager.cc:4872] TS 0f36c4166db84a3ba9aa5475377e7927 (127.0.103.3:43745): tablet 2da98881d597448e91a84719549d486d (table test-workload [id=8ce889aa53fa43c1984f8b00a4a8ef0a]) successfully deleted
I20250111 02:53:27.550278 5024 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:27.551039 5024 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:27.551182 5024 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:29.551537 5024 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:29.552206 5024 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:29.552331 5024 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:31.552726 5024 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:31.553442 5024 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:31.553586 5024 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:33.553894 5024 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:33.554550 5024 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:33.554673 5024 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:33.975113 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:53:33.991720 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:33.992043 412 tablet_replica.cc:331] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c: stopping tablet replica
I20250111 02:53:33.992247 412 raft_consensus.cc:2238] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:33.992568 412 raft_consensus.cc:2267] T 2da98881d597448e91a84719549d486d P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:33.994190 412 tablet_replica.cc:331] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c: stopping tablet replica
I20250111 02:53:33.994343 412 raft_consensus.cc:2238] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:33.994515 412 raft_consensus.cc:2267] T 3278cb6291b04b9792838f01e82bb2e3 P 4e713206f94f4883baf769d03ca2162c [term 1 FOLLOWER]: Raft consensus is shut down!
W20250111 02:53:34.001009 5114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.1:36929: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250111 02:53:34.001967 5114 consensus_peers.cc:487] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 -> Peer 4e713206f94f4883baf769d03ca2162c (127.0.103.1:36929): Couldn't send request to peer 4e713206f94f4883baf769d03ca2162c. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:36929: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:34.009616 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:53:34.012780 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
W20250111 02:53:34.017910 5187 consensus_peers.cc:487] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 -> Peer 45dc59c61cbc42dfb01fcbd99027a196 (127.0.103.2:42467): Couldn't send request to peer 45dc59c61cbc42dfb01fcbd99027a196. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:34.022333 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:34.022605 412 tablet_replica.cc:331] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196: stopping tablet replica
I20250111 02:53:34.022782 412 raft_consensus.cc:2238] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:34.023018 412 raft_consensus.cc:2267] T 2da98881d597448e91a84719549d486d P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:34.023586 412 tablet_replica.cc:331] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196: stopping tablet replica
I20250111 02:53:34.023695 412 raft_consensus.cc:2238] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:34.023928 412 raft_consensus.cc:2267] T 3278cb6291b04b9792838f01e82bb2e3 P 45dc59c61cbc42dfb01fcbd99027a196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:34.024325 412 tablet_replica.cc:331] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196: stopping tablet replica
I20250111 02:53:34.024427 412 raft_consensus.cc:2238] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:34.024662 412 raft_consensus.cc:2267] T 0db884613c4746c79e38a78d7c3b050f P 45dc59c61cbc42dfb01fcbd99027a196 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:34.038668 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:53:34.041893 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:53:34.066890 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:34.067217 412 tablet_replica.cc:331] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927: stopping tablet replica
I20250111 02:53:34.067376 412 raft_consensus.cc:2238] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:53:34.067685 412 raft_consensus.cc:2267] T 0db884613c4746c79e38a78d7c3b050f P 0f36c4166db84a3ba9aa5475377e7927 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:34.068234 412 tablet_replica.cc:331] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927: stopping tablet replica
I20250111 02:53:34.068352 412 raft_consensus.cc:2238] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:34.068471 412 raft_consensus.cc:2267] T 3278cb6291b04b9792838f01e82bb2e3 P 0f36c4166db84a3ba9aa5475377e7927 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:34.082264 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:53:34.085574 412 tablet_server.cc:178] TabletServer@127.0.103.4:0 shutting down...
I20250111 02:53:34.093375 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:34.093612 412 tablet_replica.cc:331] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd: stopping tablet replica
I20250111 02:53:34.093771 412 raft_consensus.cc:2238] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:34.094020 412 raft_consensus.cc:2267] T 2da98881d597448e91a84719549d486d P a99b5d3fae6040afa0fd48c6340ce0bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:34.094586 412 tablet_replica.cc:331] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd: stopping tablet replica
I20250111 02:53:34.094718 412 raft_consensus.cc:2238] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:34.094882 412 raft_consensus.cc:2267] T 0db884613c4746c79e38a78d7c3b050f P a99b5d3fae6040afa0fd48c6340ce0bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:34.107990 412 tablet_server.cc:195] TabletServer@127.0.103.4:0 shutdown complete.
I20250111 02:53:34.110878 412 master.cc:537] Master@127.0.103.62:33129 shutting down...
I20250111 02:53:34.118408 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:34.118636 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:34.118729 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P d17c3e33fb5e48998e85958805c00969: stopping tablet replica
I20250111 02:53:34.131780 412 master.cc:559] Master@127.0.103.62:33129 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10666 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20250111 02:53:34.144722 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:38755
I20250111 02:53:34.145182 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:34.147958 5419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:34.148469 5420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:34.149453 5422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:34.150578 412 server_base.cc:1034] running on GCE node
I20250111 02:53:34.150821 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:34.150866 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:34.150893 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564014150890 us; error 0 us; skew 500 ppm
I20250111 02:53:34.151001 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:34.157078 412 webserver.cc:458] Webserver started at http://127.0.103.62:45877/ using document root <none> and password file <none>
I20250111 02:53:34.157291 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:34.157346 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:34.157444 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:34.158088 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "abeee41d3df8450da1b5bd7b4f7ac75a"
format_stamp: "Formatted at 2025-01-11 02:53:34 on dist-test-slave-4xrj"
I20250111 02:53:34.160605 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:34.162077 5427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.162387 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:34.162482 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "abeee41d3df8450da1b5bd7b4f7ac75a"
format_stamp: "Formatted at 2025-01-11 02:53:34 on dist-test-slave-4xrj"
I20250111 02:53:34.162592 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:34.182301 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:34.182844 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:34.193428 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:38755
I20250111 02:53:34.193444 5478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:38755 every 8 connection(s)
I20250111 02:53:34.194660 5479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:34.198405 5479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a: Bootstrap starting.
I20250111 02:53:34.199636 5479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:34.201052 5479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a: No bootstrap required, opened a new log
I20250111 02:53:34.201622 5479 raft_consensus.cc:357] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abeee41d3df8450da1b5bd7b4f7ac75a" member_type: VOTER }
I20250111 02:53:34.201735 5479 raft_consensus.cc:383] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:34.201794 5479 raft_consensus.cc:738] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abeee41d3df8450da1b5bd7b4f7ac75a, State: Initialized, Role: FOLLOWER
I20250111 02:53:34.202019 5479 consensus_queue.cc:260] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abeee41d3df8450da1b5bd7b4f7ac75a" member_type: VOTER }
I20250111 02:53:34.202129 5479 raft_consensus.cc:397] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:53:34.202186 5479 raft_consensus.cc:491] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:53:34.202261 5479 raft_consensus.cc:3054] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:34.203778 5479 raft_consensus.cc:513] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abeee41d3df8450da1b5bd7b4f7ac75a" member_type: VOTER }
I20250111 02:53:34.203964 5479 leader_election.cc:304] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abeee41d3df8450da1b5bd7b4f7ac75a; no voters:
I20250111 02:53:34.204294 5479 leader_election.cc:290] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:53:34.204366 5482 raft_consensus.cc:2798] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:34.204905 5482 raft_consensus.cc:695] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 1 LEADER]: Becoming Leader. State: Replica: abeee41d3df8450da1b5bd7b4f7ac75a, State: Running, Role: LEADER
I20250111 02:53:34.205121 5479 sys_catalog.cc:564] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:53:34.205123 5482 consensus_queue.cc:237] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abeee41d3df8450da1b5bd7b4f7ac75a" member_type: VOTER }
I20250111 02:53:34.205878 5484 sys_catalog.cc:455] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [sys.catalog]: SysCatalogTable state changed. Reason: New leader abeee41d3df8450da1b5bd7b4f7ac75a. Latest consensus state: current_term: 1 leader_uuid: "abeee41d3df8450da1b5bd7b4f7ac75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abeee41d3df8450da1b5bd7b4f7ac75a" member_type: VOTER } }
I20250111 02:53:34.206066 5484 sys_catalog.cc:458] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [sys.catalog]: This master's current role is: LEADER
I20250111 02:53:34.206087 5483 sys_catalog.cc:455] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abeee41d3df8450da1b5bd7b4f7ac75a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abeee41d3df8450da1b5bd7b4f7ac75a" member_type: VOTER } }
I20250111 02:53:34.206375 5483 sys_catalog.cc:458] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [sys.catalog]: This master's current role is: LEADER
I20250111 02:53:34.207449 5487 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:53:34.208557 5487 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:53:34.211575 5487 catalog_manager.cc:1348] Generated new cluster ID: 859c84912ae44d72aba5d0915462c1b7
I20250111 02:53:34.211661 5487 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:53:34.214761 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:53:34.223425 5487 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:53:34.224215 5487 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:53:34.232537 5487 catalog_manager.cc:5899] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a: Generated new TSK 0
I20250111 02:53:34.232745 5487 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:53:34.247365 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:34.250177 5500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:34.250852 5501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:34.251884 5503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:34.252727 412 server_base.cc:1034] running on GCE node
I20250111 02:53:34.252964 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:34.253019 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:34.253062 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564014253047 us; error 0 us; skew 500 ppm
I20250111 02:53:34.253180 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:34.254390 412 webserver.cc:458] Webserver started at http://127.0.103.1:41043/ using document root <none> and password file <none>
I20250111 02:53:34.254581 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:34.254647 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:34.254758 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:34.255369 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "4fa864fc82904c379dfc77bddfa760dd"
format_stamp: "Formatted at 2025-01-11 02:53:34 on dist-test-slave-4xrj"
I20250111 02:53:34.257769 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:53:34.259328 5508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.259687 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:53:34.259784 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "4fa864fc82904c379dfc77bddfa760dd"
format_stamp: "Formatted at 2025-01-11 02:53:34 on dist-test-slave-4xrj"
I20250111 02:53:34.259904 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:34.271926 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:34.272408 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:34.272909 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:34.290642 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:34.290745 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.290804 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:34.290835 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.301986 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:44025
I20250111 02:53:34.302008 5570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:44025 every 8 connection(s)
I20250111 02:53:34.303494 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:34.306300 5575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:34.306924 5576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:34.307199 412 server_base.cc:1034] running on GCE node
W20250111 02:53:34.308174 5578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:34.308547 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:34.308607 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:34.308641 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564014308636 us; error 0 us; skew 500 ppm
I20250111 02:53:34.308808 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:34.310030 5571 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38755
I20250111 02:53:34.310127 412 webserver.cc:458] Webserver started at http://127.0.103.2:45081/ using document root <none> and password file <none>
I20250111 02:53:34.310175 5571 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:34.310333 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:34.310415 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:34.310524 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:34.310542 5571 heartbeater.cc:510] Master 127.0.103.62:38755 requested a full tablet report, sending...
I20250111 02:53:34.311157 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "1dcbcf62f7f54188ba6b6d11731eeb64"
format_stamp: "Formatted at 2025-01-11 02:53:34 on dist-test-slave-4xrj"
I20250111 02:53:34.311420 5444 ts_manager.cc:194] Registered new tserver with Master: 4fa864fc82904c379dfc77bddfa760dd (127.0.103.1:44025)
I20250111 02:53:34.312889 5444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35898
I20250111 02:53:34.313998 412 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250111 02:53:34.315349 5583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.315619 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:34.315707 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "1dcbcf62f7f54188ba6b6d11731eeb64"
format_stamp: "Formatted at 2025-01-11 02:53:34 on dist-test-slave-4xrj"
I20250111 02:53:34.315809 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:34.328003 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:34.328478 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:34.328962 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:34.329634 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:34.329686 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.329751 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:34.329790 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.340711 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:41451
I20250111 02:53:34.340734 5645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:41451 every 8 connection(s)
I20250111 02:53:34.342132 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:34.344799 5649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:34.345642 5650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:34.345903 5652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:34.346899 412 server_base.cc:1034] running on GCE node
I20250111 02:53:34.347136 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:34.347183 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:34.347206 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564014347203 us; error 0 us; skew 500 ppm
I20250111 02:53:34.347323 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:34.348372 5646 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38755
I20250111 02:53:34.348484 5646 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:34.348680 412 webserver.cc:458] Webserver started at http://127.0.103.3:35407/ using document root <none> and password file <none>
I20250111 02:53:34.348819 5646 heartbeater.cc:510] Master 127.0.103.62:38755 requested a full tablet report, sending...
I20250111 02:53:34.348861 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:34.348922 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:34.349025 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:34.349567 5444 ts_manager.cc:194] Registered new tserver with Master: 1dcbcf62f7f54188ba6b6d11731eeb64 (127.0.103.2:41451)
I20250111 02:53:34.349642 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "29b1c7604c3e476383b2d0b48f8bd498"
format_stamp: "Formatted at 2025-01-11 02:53:34 on dist-test-slave-4xrj"
I20250111 02:53:34.350247 5444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35910
I20250111 02:53:34.351868 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250111 02:53:34.353236 5657 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.353518 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:53:34.353618 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "29b1c7604c3e476383b2d0b48f8bd498"
format_stamp: "Formatted at 2025-01-11 02:53:34 on dist-test-slave-4xrj"
I20250111 02:53:34.353706 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:34.369721 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:34.370186 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:34.370672 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:34.371398 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:34.371450 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.371520 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:34.371558 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:34.382944 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:37209
I20250111 02:53:34.382982 5719 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:37209 every 8 connection(s)
I20250111 02:53:34.388742 5720 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38755
I20250111 02:53:34.388859 5720 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:34.389127 5720 heartbeater.cc:510] Master 127.0.103.62:38755 requested a full tablet report, sending...
I20250111 02:53:34.389823 5444 ts_manager.cc:194] Registered new tserver with Master: 29b1c7604c3e476383b2d0b48f8bd498 (127.0.103.3:37209)
I20250111 02:53:34.390475 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006721695s
I20250111 02:53:34.390522 5444 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35926
I20250111 02:53:35.314235 5571 heartbeater.cc:502] Master 127.0.103.62:38755 was elected leader, sending a full tablet report...
I20250111 02:53:35.351284 5646 heartbeater.cc:502] Master 127.0.103.62:38755 was elected leader, sending a full tablet report...
I20250111 02:53:35.391652 5720 heartbeater.cc:502] Master 127.0.103.62:38755 was elected leader, sending a full tablet report...
I20250111 02:53:35.418005 412 test_util.cc:274] Using random seed: 1363462780
I20250111 02:53:35.431425 5444 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53334:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:53:35.431991 5444 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:53:35.446676 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 2161d47983634ced92a9f21878b18d37 (DEFAULT_TABLE table=test-workload [id=5a1669ccce6c48c5bf3610c1514c31b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:53:35.447144 5685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2161d47983634ced92a9f21878b18d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:35.447098 5536 tablet_service.cc:1467] Processing CreateTablet for tablet 2161d47983634ced92a9f21878b18d37 (DEFAULT_TABLE table=test-workload [id=5a1669ccce6c48c5bf3610c1514c31b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:53:35.447162 5611 tablet_service.cc:1467] Processing CreateTablet for tablet 2161d47983634ced92a9f21878b18d37 (DEFAULT_TABLE table=test-workload [id=5a1669ccce6c48c5bf3610c1514c31b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250111 02:53:35.447500 5536 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2161d47983634ced92a9f21878b18d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:35.447500 5611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2161d47983634ced92a9f21878b18d37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:35.451968 5740 tablet_bootstrap.cc:492] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498: Bootstrap starting.
I20250111 02:53:35.453691 5740 tablet_bootstrap.cc:654] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:35.455418 5740 tablet_bootstrap.cc:492] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498: No bootstrap required, opened a new log
I20250111 02:53:35.455540 5740 ts_tablet_manager.cc:1397] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:53:35.456115 5740 raft_consensus.cc:357] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.456269 5740 raft_consensus.cc:383] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:35.456326 5740 raft_consensus.cc:738] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b1c7604c3e476383b2d0b48f8bd498, State: Initialized, Role: FOLLOWER
I20250111 02:53:35.456499 5740 consensus_queue.cc:260] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.457098 5740 ts_tablet_manager.cc:1428] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250111 02:53:35.457487 5741 tablet_bootstrap.cc:492] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64: Bootstrap starting.
I20250111 02:53:35.457998 5742 tablet_bootstrap.cc:492] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd: Bootstrap starting.
I20250111 02:53:35.459637 5741 tablet_bootstrap.cc:654] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:35.459637 5742 tablet_bootstrap.cc:654] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:35.461583 5741 tablet_bootstrap.cc:492] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64: No bootstrap required, opened a new log
I20250111 02:53:35.461583 5742 tablet_bootstrap.cc:492] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd: No bootstrap required, opened a new log
I20250111 02:53:35.461750 5742 ts_tablet_manager.cc:1397] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:53:35.461750 5741 ts_tablet_manager.cc:1397] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:53:35.462555 5742 raft_consensus.cc:357] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.462555 5741 raft_consensus.cc:357] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.462774 5742 raft_consensus.cc:383] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:35.462790 5741 raft_consensus.cc:383] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:35.462837 5742 raft_consensus.cc:738] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa864fc82904c379dfc77bddfa760dd, State: Initialized, Role: FOLLOWER
I20250111 02:53:35.462841 5741 raft_consensus.cc:738] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dcbcf62f7f54188ba6b6d11731eeb64, State: Initialized, Role: FOLLOWER
I20250111 02:53:35.463069 5742 consensus_queue.cc:260] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.463069 5741 consensus_queue.cc:260] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.464128 5741 ts_tablet_manager.cc:1428] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:35.464813 5742 ts_tablet_manager.cc:1428] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:53:35.550860 5744 raft_consensus.cc:491] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:53:35.551054 5744 raft_consensus.cc:513] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.551790 5744 leader_election.cc:290] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1dcbcf62f7f54188ba6b6d11731eeb64 (127.0.103.2:41451), 4fa864fc82904c379dfc77bddfa760dd (127.0.103.1:44025)
I20250111 02:53:35.557709 5621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2161d47983634ced92a9f21878b18d37" candidate_uuid: "29b1c7604c3e476383b2d0b48f8bd498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" is_pre_election: true
I20250111 02:53:35.557709 5546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2161d47983634ced92a9f21878b18d37" candidate_uuid: "29b1c7604c3e476383b2d0b48f8bd498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa864fc82904c379dfc77bddfa760dd" is_pre_election: true
I20250111 02:53:35.557989 5621 raft_consensus.cc:2463] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b1c7604c3e476383b2d0b48f8bd498 in term 0.
I20250111 02:53:35.558015 5546 raft_consensus.cc:2463] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29b1c7604c3e476383b2d0b48f8bd498 in term 0.
I20250111 02:53:35.558405 5659 leader_election.cc:304] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1dcbcf62f7f54188ba6b6d11731eeb64, 29b1c7604c3e476383b2d0b48f8bd498, 4fa864fc82904c379dfc77bddfa760dd; no voters:
I20250111 02:53:35.558660 5744 raft_consensus.cc:2798] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:53:35.558755 5744 raft_consensus.cc:491] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:53:35.558833 5744 raft_consensus.cc:3054] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:35.560524 5744 raft_consensus.cc:513] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.561086 5744 leader_election.cc:290] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [CANDIDATE]: Term 1 election: Requested vote from peers 1dcbcf62f7f54188ba6b6d11731eeb64 (127.0.103.2:41451), 4fa864fc82904c379dfc77bddfa760dd (127.0.103.1:44025)
I20250111 02:53:35.561419 5546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2161d47983634ced92a9f21878b18d37" candidate_uuid: "29b1c7604c3e476383b2d0b48f8bd498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fa864fc82904c379dfc77bddfa760dd"
I20250111 02:53:35.561498 5621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2161d47983634ced92a9f21878b18d37" candidate_uuid: "29b1c7604c3e476383b2d0b48f8bd498" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64"
I20250111 02:53:35.561635 5546 raft_consensus.cc:3054] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:35.561638 5621 raft_consensus.cc:3054] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:35.563325 5546 raft_consensus.cc:2463] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b1c7604c3e476383b2d0b48f8bd498 in term 1.
I20250111 02:53:35.563407 5621 raft_consensus.cc:2463] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29b1c7604c3e476383b2d0b48f8bd498 in term 1.
I20250111 02:53:35.563678 5661 leader_election.cc:304] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29b1c7604c3e476383b2d0b48f8bd498, 4fa864fc82904c379dfc77bddfa760dd; no voters:
I20250111 02:53:35.563918 5744 raft_consensus.cc:2798] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:35.564357 5744 raft_consensus.cc:695] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 1 LEADER]: Becoming Leader. State: Replica: 29b1c7604c3e476383b2d0b48f8bd498, State: Running, Role: LEADER
I20250111 02:53:35.564610 5744 consensus_queue.cc:237] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } }
I20250111 02:53:35.566918 5443 catalog_manager.cc:5526] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29b1c7604c3e476383b2d0b48f8bd498 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "29b1c7604c3e476383b2d0b48f8bd498" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29b1c7604c3e476383b2d0b48f8bd498" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fa864fc82904c379dfc77bddfa760dd" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 44025 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:35.591771 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:53:35.598785 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:35.599174 412 tablet_replica.cc:331] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd: stopping tablet replica
I20250111 02:53:35.599331 412 raft_consensus.cc:2238] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:35.599481 412 raft_consensus.cc:2267] T 2161d47983634ced92a9f21878b18d37 P 4fa864fc82904c379dfc77bddfa760dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:35.602804 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
W20250111 02:53:36.056994 5661 consensus_peers.cc:487] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 -> Peer 4fa864fc82904c379dfc77bddfa760dd (127.0.103.1:44025): Couldn't send request to peer 4fa864fc82904c379dfc77bddfa760dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:44025: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:36.092964 5744 consensus_queue.cc:1035] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1dcbcf62f7f54188ba6b6d11731eeb64" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 41451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:53:36.214728 5498 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:36.215101 5498 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:36.215201 5498 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20250111 02:53:38.215520 5498 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:38.215901 5498 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:38.215994 5498 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250111 02:53:38.692505 5661 consensus_peers.cc:487] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 -> Peer 4fa864fc82904c379dfc77bddfa760dd (127.0.103.1:44025): Couldn't send request to peer 4fa864fc82904c379dfc77bddfa760dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:44025: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250111 02:53:39.313665 5661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.1:44025: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20250111 02:53:40.216396 5498 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:40.216804 5498 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:40.216933 5498 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250111 02:53:41.187331 5661 consensus_peers.cc:487] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 -> Peer 4fa864fc82904c379dfc77bddfa760dd (127.0.103.1:44025): Couldn't send request to peer 4fa864fc82904c379dfc77bddfa760dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:44025: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250111 02:53:42.217356 5498 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:42.217823 5498 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:42.217962 5498 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250111 02:53:43.609812 5661 consensus_peers.cc:487] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 -> Peer 4fa864fc82904c379dfc77bddfa760dd (127.0.103.1:44025): Couldn't send request to peer 4fa864fc82904c379dfc77bddfa760dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:44025: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250111 02:53:44.218320 5498 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:44.218787 5498 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:44.218909 5498 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250111 02:53:44.766981 5661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.1:44025: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250111 02:53:46.219260 5498 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250111 02:53:46.219800 5498 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250111 02:53:46.219918 5498 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20250111 02:53:46.370415 5661 consensus_peers.cc:487] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 -> Peer 4fa864fc82904c379dfc77bddfa760dd (127.0.103.1:44025): Couldn't send request to peer 4fa864fc82904c379dfc77bddfa760dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:44025: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250111 02:53:46.659888 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:53:46.679921 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:46.680277 412 tablet_replica.cc:331] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64: stopping tablet replica
I20250111 02:53:46.680449 412 raft_consensus.cc:2238] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:46.680660 412 raft_consensus.cc:2267] T 2161d47983634ced92a9f21878b18d37 P 1dcbcf62f7f54188ba6b6d11731eeb64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:46.695993 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:53:46.698855 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:53:46.706247 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:46.706548 412 tablet_replica.cc:331] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498: stopping tablet replica
I20250111 02:53:46.706763 412 raft_consensus.cc:2238] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:46.707100 412 raft_consensus.cc:2267] T 2161d47983634ced92a9f21878b18d37 P 29b1c7604c3e476383b2d0b48f8bd498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:46.720395 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:53:46.723263 412 master.cc:537] Master@127.0.103.62:38755 shutting down...
I20250111 02:53:46.744658 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:46.744928 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:46.745045 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P abeee41d3df8450da1b5bd7b4f7ac75a: stopping tablet replica
I20250111 02:53:46.758509 412 master.cc:559] Master@127.0.103.62:38755 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12624 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20250111 02:53:46.769155 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:35083
I20250111 02:53:46.769599 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:46.772501 5785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:46.772717 5784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:46.773941 5787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:46.774940 412 server_base.cc:1034] running on GCE node
I20250111 02:53:46.775194 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:46.775241 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:46.775271 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564026775269 us; error 0 us; skew 500 ppm
I20250111 02:53:46.775393 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:46.782132 412 webserver.cc:458] Webserver started at http://127.0.103.62:41127/ using document root <none> and password file <none>
I20250111 02:53:46.782332 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:46.782385 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:46.782473 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:46.783046 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "47efffa89b6a4a41bf34d8161dd1cb77"
format_stamp: "Formatted at 2025-01-11 02:53:46 on dist-test-slave-4xrj"
I20250111 02:53:46.785410 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:53:46.786996 5792 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:46.787307 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:46.787410 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "47efffa89b6a4a41bf34d8161dd1cb77"
format_stamp: "Formatted at 2025-01-11 02:53:46 on dist-test-slave-4xrj"
I20250111 02:53:46.787539 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:46.794323 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:46.794791 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:46.805169 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:35083
I20250111 02:53:46.805188 5843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:35083 every 8 connection(s)
I20250111 02:53:46.806416 5844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:46.810060 5844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77: Bootstrap starting.
I20250111 02:53:46.811260 5844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:46.812709 5844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77: No bootstrap required, opened a new log
I20250111 02:53:46.813287 5844 raft_consensus.cc:357] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47efffa89b6a4a41bf34d8161dd1cb77" member_type: VOTER }
I20250111 02:53:46.813405 5844 raft_consensus.cc:383] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:46.813464 5844 raft_consensus.cc:738] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47efffa89b6a4a41bf34d8161dd1cb77, State: Initialized, Role: FOLLOWER
I20250111 02:53:46.813666 5844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47efffa89b6a4a41bf34d8161dd1cb77" member_type: VOTER }
I20250111 02:53:46.813787 5844 raft_consensus.cc:397] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:53:46.813871 5844 raft_consensus.cc:491] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:53:46.813957 5844 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:46.815526 5844 raft_consensus.cc:513] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47efffa89b6a4a41bf34d8161dd1cb77" member_type: VOTER }
I20250111 02:53:46.815693 5844 leader_election.cc:304] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47efffa89b6a4a41bf34d8161dd1cb77; no voters:
I20250111 02:53:46.816063 5844 leader_election.cc:290] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:53:46.816152 5847 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:46.816593 5847 raft_consensus.cc:695] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 1 LEADER]: Becoming Leader. State: Replica: 47efffa89b6a4a41bf34d8161dd1cb77, State: Running, Role: LEADER
I20250111 02:53:46.816831 5847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47efffa89b6a4a41bf34d8161dd1cb77" member_type: VOTER }
I20250111 02:53:46.816869 5844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:53:46.817636 5848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47efffa89b6a4a41bf34d8161dd1cb77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47efffa89b6a4a41bf34d8161dd1cb77" member_type: VOTER } }
I20250111 02:53:46.817749 5849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47efffa89b6a4a41bf34d8161dd1cb77. Latest consensus state: current_term: 1 leader_uuid: "47efffa89b6a4a41bf34d8161dd1cb77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47efffa89b6a4a41bf34d8161dd1cb77" member_type: VOTER } }
I20250111 02:53:46.817942 5848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [sys.catalog]: This master's current role is: LEADER
I20250111 02:53:46.818045 5849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [sys.catalog]: This master's current role is: LEADER
I20250111 02:53:46.819065 5853 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:53:46.820431 5853 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:53:46.821813 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:53:46.823091 5853 catalog_manager.cc:1348] Generated new cluster ID: e18c1029a3b3401f96b56d19b2655cb8
I20250111 02:53:46.823164 5853 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:53:46.847220 5853 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:53:46.847959 5853 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:53:46.857189 5853 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77: Generated new TSK 0
I20250111 02:53:46.857400 5853 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:53:46.886902 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:46.889724 5865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:46.890444 5866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:46.891538 5868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:46.892203 412 server_base.cc:1034] running on GCE node
I20250111 02:53:46.892465 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:46.892530 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:46.892575 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564026892570 us; error 0 us; skew 500 ppm
I20250111 02:53:46.892722 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:46.893993 412 webserver.cc:458] Webserver started at http://127.0.103.1:38589/ using document root <none> and password file <none>
I20250111 02:53:46.894214 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:46.894279 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:46.894415 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:46.895040 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "1eeddf93653e4453847337013c1f19f9"
format_stamp: "Formatted at 2025-01-11 02:53:46 on dist-test-slave-4xrj"
I20250111 02:53:46.897374 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:53:46.898876 5873 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:46.899224 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:46.899298 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "1eeddf93653e4453847337013c1f19f9"
format_stamp: "Formatted at 2025-01-11 02:53:46 on dist-test-slave-4xrj"
I20250111 02:53:46.899399 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:46.918789 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:46.919309 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:46.919821 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:46.920709 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:46.920774 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:46.920847 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:46.920882 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:46.933169 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:39533
I20250111 02:53:46.933188 5935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:39533 every 8 connection(s)
I20250111 02:53:46.934742 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:46.937770 5940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:46.938400 5941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:46.939282 5943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:46.939611 412 server_base.cc:1034] running on GCE node
I20250111 02:53:46.939848 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:46.939926 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:46.939987 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564026939983 us; error 0 us; skew 500 ppm
I20250111 02:53:46.940127 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:46.941632 5936 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35083
I20250111 02:53:46.941673 412 webserver.cc:458] Webserver started at http://127.0.103.2:46239/ using document root <none> and password file <none>
I20250111 02:53:46.941761 5936 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:46.941919 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:46.941982 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:46.942098 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:46.942130 5936 heartbeater.cc:510] Master 127.0.103.62:35083 requested a full tablet report, sending...
I20250111 02:53:46.942811 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "ee77e10db69e4f10a9bbae06ddbb112b"
format_stamp: "Formatted at 2025-01-11 02:53:46 on dist-test-slave-4xrj"
I20250111 02:53:46.942992 5809 ts_manager.cc:194] Registered new tserver with Master: 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533)
I20250111 02:53:46.944492 5809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48180
I20250111 02:53:46.945803 412 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250111 02:53:46.947225 5948 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:46.947504 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:53:46.947597 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "ee77e10db69e4f10a9bbae06ddbb112b"
format_stamp: "Formatted at 2025-01-11 02:53:46 on dist-test-slave-4xrj"
I20250111 02:53:46.947702 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:46.960896 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:46.961427 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:46.961999 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:46.962747 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:46.962803 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:46.962884 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:46.962929 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:46.974331 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:46259
I20250111 02:53:46.974352 6010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:46259 every 8 connection(s)
I20250111 02:53:46.975826 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:46.978739 6014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:46.979430 6015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:46.980563 6017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:46.981720 412 server_base.cc:1034] running on GCE node
I20250111 02:53:46.981998 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:46.982060 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:46.982110 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564026982105 us; error 0 us; skew 500 ppm
I20250111 02:53:46.982259 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:46.982275 6011 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35083
I20250111 02:53:46.982399 6011 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:46.982818 6011 heartbeater.cc:510] Master 127.0.103.62:35083 requested a full tablet report, sending...
I20250111 02:53:46.983528 412 webserver.cc:458] Webserver started at http://127.0.103.3:40167/ using document root <none> and password file <none>
I20250111 02:53:46.983724 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:46.983692 5809 ts_manager.cc:194] Registered new tserver with Master: ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2:46259)
I20250111 02:53:46.983805 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:46.983914 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:46.984478 5809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48194
I20250111 02:53:46.984491 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "c067bca729e146388673a4278dbc5b4c"
format_stamp: "Formatted at 2025-01-11 02:53:46 on dist-test-slave-4xrj"
I20250111 02:53:46.986937 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:46.988333 6022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:46.988636 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:53:46.988737 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "c067bca729e146388673a4278dbc5b4c"
format_stamp: "Formatted at 2025-01-11 02:53:46 on dist-test-slave-4xrj"
I20250111 02:53:46.988847 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:47.004572 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:47.005084 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:47.005591 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:47.006405 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:47.006461 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:47.006527 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:47.006572 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:47.017549 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:33649
I20250111 02:53:47.017572 6084 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:33649 every 8 connection(s)
I20250111 02:53:47.024168 6085 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35083
I20250111 02:53:47.024295 6085 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:47.024650 6085 heartbeater.cc:510] Master 127.0.103.62:35083 requested a full tablet report, sending...
I20250111 02:53:47.025449 5809 ts_manager.cc:194] Registered new tserver with Master: c067bca729e146388673a4278dbc5b4c (127.0.103.3:33649)
I20250111 02:53:47.026186 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007820677s
I20250111 02:53:47.026193 5809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48196
I20250111 02:53:47.945712 5936 heartbeater.cc:502] Master 127.0.103.62:35083 was elected leader, sending a full tablet report...
I20250111 02:53:47.985621 6011 heartbeater.cc:502] Master 127.0.103.62:35083 was elected leader, sending a full tablet report...
I20250111 02:53:48.027560 6085 heartbeater.cc:502] Master 127.0.103.62:35083 was elected leader, sending a full tablet report...
I20250111 02:53:48.053956 412 test_util.cc:274] Using random seed: 1376098749
I20250111 02:53:48.071455 5809 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48202:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:53:48.072127 5809 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:53:48.099215 5901 tablet_service.cc:1467] Processing CreateTablet for tablet c15e3bbee23f43cd8b81c5d057ad7b32 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:53:48.099571 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 394a0b54dc6c458494b058a2c9547d20 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:53:48.099571 5976 tablet_service.cc:1467] Processing CreateTablet for tablet c15e3bbee23f43cd8b81c5d057ad7b32 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:53:48.099745 5901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c15e3bbee23f43cd8b81c5d057ad7b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.099920 5973 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef1f1786674425a8ef7ad9a609eeac5 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:53:48.100037 5975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394a0b54dc6c458494b058a2c9547d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.100229 5976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c15e3bbee23f43cd8b81c5d057ad7b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.100478 5973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ef1f1786674425a8ef7ad9a609eeac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.102156 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 186ab318f3264d33ab17da2af5d96348 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:53:48.102502 5974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186ab318f3264d33ab17da2af5d96348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.109035 6106 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9: Bootstrap starting.
I20250111 02:53:48.109354 5899 tablet_service.cc:1467] Processing CreateTablet for tablet 186ab318f3264d33ab17da2af5d96348 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:53:48.109398 5900 tablet_service.cc:1467] Processing CreateTablet for tablet 394a0b54dc6c458494b058a2c9547d20 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:53:48.109771 5899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186ab318f3264d33ab17da2af5d96348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.111265 6106 tablet_bootstrap.cc:654] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.111361 5900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394a0b54dc6c458494b058a2c9547d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.111932 6105 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b: Bootstrap starting.
I20250111 02:53:48.113638 6105 tablet_bootstrap.cc:654] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.116005 5898 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef1f1786674425a8ef7ad9a609eeac5 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:53:48.116523 5898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ef1f1786674425a8ef7ad9a609eeac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.130239 6106 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9: No bootstrap required, opened a new log
I20250111 02:53:48.130143 6050 tablet_service.cc:1467] Processing CreateTablet for tablet c15e3bbee23f43cd8b81c5d057ad7b32 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:53:48.130281 6048 tablet_service.cc:1467] Processing CreateTablet for tablet 186ab318f3264d33ab17da2af5d96348 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:53:48.130421 6106 ts_tablet_manager.cc:1397] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.010s
I20250111 02:53:48.130671 6050 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c15e3bbee23f43cd8b81c5d057ad7b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.130784 6047 tablet_service.cc:1467] Processing CreateTablet for tablet 8ef1f1786674425a8ef7ad9a609eeac5 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:53:48.130877 6048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186ab318f3264d33ab17da2af5d96348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.131125 6047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ef1f1786674425a8ef7ad9a609eeac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.131273 6106 raft_consensus.cc:357] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.131501 6106 raft_consensus.cc:383] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.131578 6106 raft_consensus.cc:738] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eeddf93653e4453847337013c1f19f9, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.131829 6106 consensus_queue.cc:260] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.134984 6049 tablet_service.cc:1467] Processing CreateTablet for tablet 394a0b54dc6c458494b058a2c9547d20 (DEFAULT_TABLE table=test-workload [id=5001a8d5e9ac49538de667f4e02545e9]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:53:48.135401 6049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394a0b54dc6c458494b058a2c9547d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:48.138906 6105 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b: No bootstrap required, opened a new log
I20250111 02:53:48.139145 6105 ts_tablet_manager.cc:1397] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.016s
I20250111 02:53:48.140908 6105 raft_consensus.cc:357] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.142199 6105 raft_consensus.cc:383] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.142706 6105 raft_consensus.cc:738] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee77e10db69e4f10a9bbae06ddbb112b, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.143396 6105 consensus_queue.cc:260] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.146701 6106 ts_tablet_manager.cc:1428] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9: Time spent starting tablet: real 0.016s user 0.014s sys 0.002s
I20250111 02:53:48.147029 6106 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9: Bootstrap starting.
I20250111 02:53:48.147811 6105 ts_tablet_manager.cc:1428] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20250111 02:53:48.148414 6105 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b: Bootstrap starting.
I20250111 02:53:48.148963 6106 tablet_bootstrap.cc:654] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.150344 6110 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c: Bootstrap starting.
I20250111 02:53:48.152707 6106 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9: No bootstrap required, opened a new log
I20250111 02:53:48.152757 6110 tablet_bootstrap.cc:654] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.152868 6106 ts_tablet_manager.cc:1397] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250111 02:53:48.153146 6105 tablet_bootstrap.cc:654] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.153787 6106 raft_consensus.cc:357] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.154090 6106 raft_consensus.cc:383] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.154207 6106 raft_consensus.cc:738] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eeddf93653e4453847337013c1f19f9, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.154531 6110 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c: No bootstrap required, opened a new log
I20250111 02:53:48.154673 6110 ts_tablet_manager.cc:1397] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:53:48.154592 6106 consensus_queue.cc:260] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.155354 6106 ts_tablet_manager.cc:1428] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250111 02:53:48.155541 6110 raft_consensus.cc:357] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.155706 6106 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9: Bootstrap starting.
I20250111 02:53:48.155761 6110 raft_consensus.cc:383] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.155843 6110 raft_consensus.cc:738] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c067bca729e146388673a4278dbc5b4c, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.156095 6110 consensus_queue.cc:260] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.157969 6106 tablet_bootstrap.cc:654] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.160550 6110 ts_tablet_manager.cc:1428] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20250111 02:53:48.161346 6110 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c: Bootstrap starting.
I20250111 02:53:48.161607 6106 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9: No bootstrap required, opened a new log
I20250111 02:53:48.161871 6106 ts_tablet_manager.cc:1397] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20250111 02:53:48.162945 6106 raft_consensus.cc:357] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.163328 6106 raft_consensus.cc:383] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.163499 6106 raft_consensus.cc:738] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eeddf93653e4453847337013c1f19f9, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.163944 6110 tablet_bootstrap.cc:654] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.163894 6106 consensus_queue.cc:260] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.164647 6106 ts_tablet_manager.cc:1428] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250111 02:53:48.164984 6106 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9: Bootstrap starting.
I20250111 02:53:48.166152 6105 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b: No bootstrap required, opened a new log
I20250111 02:53:48.166283 6105 ts_tablet_manager.cc:1397] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20250111 02:53:48.166666 6106 tablet_bootstrap.cc:654] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.167112 6105 raft_consensus.cc:357] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.167337 6105 raft_consensus.cc:383] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.167434 6105 raft_consensus.cc:738] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee77e10db69e4f10a9bbae06ddbb112b, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.167805 6105 consensus_queue.cc:260] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.168020 6110 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c: No bootstrap required, opened a new log
I20250111 02:53:48.168169 6110 ts_tablet_manager.cc:1397] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20250111 02:53:48.168394 6105 ts_tablet_manager.cc:1428] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:48.168649 6105 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b: Bootstrap starting.
I20250111 02:53:48.168937 6110 raft_consensus.cc:357] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.169128 6110 raft_consensus.cc:383] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.169217 6110 raft_consensus.cc:738] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c067bca729e146388673a4278dbc5b4c, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.169459 6110 consensus_queue.cc:260] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.170464 6105 tablet_bootstrap.cc:654] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.170614 6110 ts_tablet_manager.cc:1428] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:48.170915 6110 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c: Bootstrap starting.
I20250111 02:53:48.173460 6110 tablet_bootstrap.cc:654] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.174589 6106 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9: No bootstrap required, opened a new log
I20250111 02:53:48.174757 6106 ts_tablet_manager.cc:1397] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20250111 02:53:48.174892 6105 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b: No bootstrap required, opened a new log
I20250111 02:53:48.175045 6105 ts_tablet_manager.cc:1397] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250111 02:53:48.175570 6106 raft_consensus.cc:357] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.175658 6105 raft_consensus.cc:357] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.175832 6105 raft_consensus.cc:383] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.175663 6110 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c: No bootstrap required, opened a new log
I20250111 02:53:48.175808 6106 raft_consensus.cc:383] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.175935 6105 raft_consensus.cc:738] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee77e10db69e4f10a9bbae06ddbb112b, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.176019 6106 raft_consensus.cc:738] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eeddf93653e4453847337013c1f19f9, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.176030 6110 ts_tablet_manager.cc:1397] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250111 02:53:48.176191 6105 consensus_queue.cc:260] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.176218 6106 consensus_queue.cc:260] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.176811 6106 ts_tablet_manager.cc:1428] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250111 02:53:48.176880 6110 raft_consensus.cc:357] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.177110 6110 raft_consensus.cc:383] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.177194 6110 raft_consensus.cc:738] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c067bca729e146388673a4278dbc5b4c, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.177369 6105 ts_tablet_manager.cc:1428] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:48.177428 6110 consensus_queue.cc:260] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.177609 6105 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b: Bootstrap starting.
I20250111 02:53:48.179227 6105 tablet_bootstrap.cc:654] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.179823 6110 ts_tablet_manager.cc:1428] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250111 02:53:48.180719 6110 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c: Bootstrap starting.
I20250111 02:53:48.180833 6105 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b: No bootstrap required, opened a new log
I20250111 02:53:48.180980 6105 ts_tablet_manager.cc:1397] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250111 02:53:48.181699 6105 raft_consensus.cc:357] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.181948 6105 raft_consensus.cc:383] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.182058 6105 raft_consensus.cc:738] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee77e10db69e4f10a9bbae06ddbb112b, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.182296 6105 consensus_queue.cc:260] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.182760 6105 ts_tablet_manager.cc:1428] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:48.182790 6110 tablet_bootstrap.cc:654] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c: Neither blocks nor log segments found. Creating new log.
I20250111 02:53:48.184561 6110 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c: No bootstrap required, opened a new log
I20250111 02:53:48.184693 6110 ts_tablet_manager.cc:1397] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250111 02:53:48.185452 6110 raft_consensus.cc:357] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.185668 6110 raft_consensus.cc:383] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:53:48.185758 6110 raft_consensus.cc:738] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c067bca729e146388673a4278dbc5b4c, State: Initialized, Role: FOLLOWER
I20250111 02:53:48.186038 6110 consensus_queue.cc:260] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.186427 6110 ts_tablet_manager.cc:1428] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:48.231243 6111 raft_consensus.cc:491] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:53:48.231443 6111 raft_consensus.cc:513] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.232189 6111 leader_election.cc:290] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533), c067bca729e146388673a4278dbc5b4c (127.0.103.3:33649)
I20250111 02:53:48.238574 5911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394a0b54dc6c458494b058a2c9547d20" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eeddf93653e4453847337013c1f19f9" is_pre_election: true
I20250111 02:53:48.238674 6060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394a0b54dc6c458494b058a2c9547d20" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c067bca729e146388673a4278dbc5b4c" is_pre_election: true
I20250111 02:53:48.238826 5911 raft_consensus.cc:2463] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 0.
I20250111 02:53:48.238914 6060 raft_consensus.cc:2463] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 0.
I20250111 02:53:48.239252 5951 leader_election.cc:304] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eeddf93653e4453847337013c1f19f9, ee77e10db69e4f10a9bbae06ddbb112b; no voters:
I20250111 02:53:48.239516 6111 raft_consensus.cc:2798] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:53:48.239612 6111 raft_consensus.cc:491] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:53:48.239698 6111 raft_consensus.cc:3054] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.241465 6111 raft_consensus.cc:513] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.241976 6111 leader_election.cc:290] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 election: Requested vote from peers 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533), c067bca729e146388673a4278dbc5b4c (127.0.103.3:33649)
I20250111 02:53:48.242342 5911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394a0b54dc6c458494b058a2c9547d20" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eeddf93653e4453847337013c1f19f9"
I20250111 02:53:48.242365 6060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "394a0b54dc6c458494b058a2c9547d20" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c067bca729e146388673a4278dbc5b4c"
I20250111 02:53:48.242513 5911 raft_consensus.cc:3054] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.242520 6060 raft_consensus.cc:3054] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.244249 5911 raft_consensus.cc:2463] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 1.
I20250111 02:53:48.244246 6060 raft_consensus.cc:2463] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 1.
I20250111 02:53:48.244624 5951 leader_election.cc:304] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eeddf93653e4453847337013c1f19f9, ee77e10db69e4f10a9bbae06ddbb112b; no voters:
I20250111 02:53:48.244870 6111 raft_consensus.cc:2798] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:48.245265 6111 raft_consensus.cc:695] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Becoming Leader. State: Replica: ee77e10db69e4f10a9bbae06ddbb112b, State: Running, Role: LEADER
I20250111 02:53:48.245533 6111 consensus_queue.cc:237] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.246220 6115 raft_consensus.cc:491] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:53:48.246357 6115 raft_consensus.cc:513] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.247210 6115 leader_election.cc:290] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533), ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2:46259)
I20250111 02:53:48.248131 5809 catalog_manager.cc:5526] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: term changed from 0 to 1, leader changed from <none> to ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2). New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:48.252797 6111 raft_consensus.cc:491] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:53:48.252938 6111 raft_consensus.cc:513] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.253440 6111 leader_election.cc:290] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533), c067bca729e146388673a4278dbc5b4c (127.0.103.3:33649)
I20250111 02:53:48.253814 6060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186ab318f3264d33ab17da2af5d96348" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c067bca729e146388673a4278dbc5b4c" is_pre_election: true
I20250111 02:53:48.254074 6060 raft_consensus.cc:2463] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 0.
I20250111 02:53:48.254076 5911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186ab318f3264d33ab17da2af5d96348" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eeddf93653e4453847337013c1f19f9" is_pre_election: true
I20250111 02:53:48.254454 5911 raft_consensus.cc:2463] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 0.
I20250111 02:53:48.254455 5951 leader_election.cc:304] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c067bca729e146388673a4278dbc5b4c, ee77e10db69e4f10a9bbae06ddbb112b; no voters:
I20250111 02:53:48.254935 6111 raft_consensus.cc:2798] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:53:48.255076 6111 raft_consensus.cc:491] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:53:48.255173 6111 raft_consensus.cc:3054] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.256968 6111 raft_consensus.cc:513] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.257464 6111 leader_election.cc:290] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 election: Requested vote from peers 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533), c067bca729e146388673a4278dbc5b4c (127.0.103.3:33649)
I20250111 02:53:48.257871 5911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186ab318f3264d33ab17da2af5d96348" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eeddf93653e4453847337013c1f19f9"
I20250111 02:53:48.258052 5911 raft_consensus.cc:3054] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.259676 5911 raft_consensus.cc:2463] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 1.
I20250111 02:53:48.258052 6060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "186ab318f3264d33ab17da2af5d96348" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c067bca729e146388673a4278dbc5b4c"
I20250111 02:53:48.259981 5951 leader_election.cc:304] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eeddf93653e4453847337013c1f19f9, ee77e10db69e4f10a9bbae06ddbb112b; no voters:
I20250111 02:53:48.260154 6060 raft_consensus.cc:3054] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.260284 6111 raft_consensus.cc:2798] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:48.260414 6111 raft_consensus.cc:695] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Becoming Leader. State: Replica: ee77e10db69e4f10a9bbae06ddbb112b, State: Running, Role: LEADER
I20250111 02:53:48.260661 6111 consensus_queue.cc:237] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.263917 6060 raft_consensus.cc:2463] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 1.
I20250111 02:53:48.266623 5809 catalog_manager.cc:5526] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: term changed from 0 to 1, leader changed from <none> to ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2). New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:48.266965 5986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15e3bbee23f43cd8b81c5d057ad7b32" candidate_uuid: "c067bca729e146388673a4278dbc5b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" is_pre_election: true
I20250111 02:53:48.267195 5986 raft_consensus.cc:2463] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c067bca729e146388673a4278dbc5b4c in term 0.
I20250111 02:53:48.267578 5911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15e3bbee23f43cd8b81c5d057ad7b32" candidate_uuid: "c067bca729e146388673a4278dbc5b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eeddf93653e4453847337013c1f19f9" is_pre_election: true
I20250111 02:53:48.267664 6024 leader_election.cc:304] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c067bca729e146388673a4278dbc5b4c, ee77e10db69e4f10a9bbae06ddbb112b; no voters:
I20250111 02:53:48.267776 5911 raft_consensus.cc:2463] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c067bca729e146388673a4278dbc5b4c in term 0.
I20250111 02:53:48.267921 6115 raft_consensus.cc:2798] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:53:48.268007 6115 raft_consensus.cc:491] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:53:48.268102 6115 raft_consensus.cc:3054] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.270257 6115 raft_consensus.cc:513] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.270802 6115 leader_election.cc:290] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [CANDIDATE]: Term 1 election: Requested vote from peers 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533), ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2:46259)
I20250111 02:53:48.271067 5911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15e3bbee23f43cd8b81c5d057ad7b32" candidate_uuid: "c067bca729e146388673a4278dbc5b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eeddf93653e4453847337013c1f19f9"
I20250111 02:53:48.271252 5911 raft_consensus.cc:3054] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.271256 5986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15e3bbee23f43cd8b81c5d057ad7b32" candidate_uuid: "c067bca729e146388673a4278dbc5b4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee77e10db69e4f10a9bbae06ddbb112b"
I20250111 02:53:48.271497 5986 raft_consensus.cc:3054] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.273792 5911 raft_consensus.cc:2463] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c067bca729e146388673a4278dbc5b4c in term 1.
I20250111 02:53:48.274040 5986 raft_consensus.cc:2463] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c067bca729e146388673a4278dbc5b4c in term 1.
I20250111 02:53:48.274255 6025 leader_election.cc:304] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eeddf93653e4453847337013c1f19f9, c067bca729e146388673a4278dbc5b4c; no voters:
I20250111 02:53:48.274497 6115 raft_consensus.cc:2798] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:48.275305 6115 raft_consensus.cc:695] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 1 LEADER]: Becoming Leader. State: Replica: c067bca729e146388673a4278dbc5b4c, State: Running, Role: LEADER
I20250111 02:53:48.275561 6115 consensus_queue.cc:237] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } }
I20250111 02:53:48.277658 5809 catalog_manager.cc:5526] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c reported cstate change: term changed from 0 to 1, leader changed from <none> to c067bca729e146388673a4278dbc5b4c (127.0.103.3). New cstate: current_term: 1 leader_uuid: "c067bca729e146388673a4278dbc5b4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:48.285073 6111 raft_consensus.cc:491] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:53:48.285252 6111 raft_consensus.cc:513] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.285880 6111 leader_election.cc:290] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533), c067bca729e146388673a4278dbc5b4c (127.0.103.3:33649)
I20250111 02:53:48.286160 5911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef1f1786674425a8ef7ad9a609eeac5" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eeddf93653e4453847337013c1f19f9" is_pre_election: true
I20250111 02:53:48.286219 6060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef1f1786674425a8ef7ad9a609eeac5" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c067bca729e146388673a4278dbc5b4c" is_pre_election: true
I20250111 02:53:48.286398 5911 raft_consensus.cc:2463] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 0.
I20250111 02:53:48.286396 6060 raft_consensus.cc:2463] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 0.
I20250111 02:53:48.286793 5951 leader_election.cc:304] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eeddf93653e4453847337013c1f19f9, ee77e10db69e4f10a9bbae06ddbb112b; no voters:
I20250111 02:53:48.287029 6111 raft_consensus.cc:2798] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:53:48.287128 6111 raft_consensus.cc:491] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:53:48.287220 6111 raft_consensus.cc:3054] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.289074 6111 raft_consensus.cc:513] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.289554 6111 leader_election.cc:290] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 election: Requested vote from peers 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533), c067bca729e146388673a4278dbc5b4c (127.0.103.3:33649)
I20250111 02:53:48.289954 6060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef1f1786674425a8ef7ad9a609eeac5" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c067bca729e146388673a4278dbc5b4c"
I20250111 02:53:48.289942 5911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ef1f1786674425a8ef7ad9a609eeac5" candidate_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eeddf93653e4453847337013c1f19f9"
I20250111 02:53:48.290146 6060 raft_consensus.cc:3054] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.290179 5911 raft_consensus.cc:3054] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:53:48.292620 5911 raft_consensus.cc:2463] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 1.
I20250111 02:53:48.292620 6060 raft_consensus.cc:2463] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee77e10db69e4f10a9bbae06ddbb112b in term 1.
I20250111 02:53:48.293006 5951 leader_election.cc:304] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eeddf93653e4453847337013c1f19f9, ee77e10db69e4f10a9bbae06ddbb112b; no voters:
I20250111 02:53:48.293234 6111 raft_consensus.cc:2798] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:53:48.293378 6111 raft_consensus.cc:695] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Becoming Leader. State: Replica: ee77e10db69e4f10a9bbae06ddbb112b, State: Running, Role: LEADER
I20250111 02:53:48.293622 6111 consensus_queue.cc:237] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } }
I20250111 02:53:48.295993 5809 catalog_manager.cc:5526] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: term changed from 0 to 1, leader changed from <none> to ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2). New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:48.331415 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:53:48.348687 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:48.349099 412 tablet_replica.cc:331] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9: stopping tablet replica
I20250111 02:53:48.349293 412 raft_consensus.cc:2238] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:48.349458 412 raft_consensus.cc:2267] T 186ab318f3264d33ab17da2af5d96348 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:48.350147 412 tablet_replica.cc:331] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9: stopping tablet replica
I20250111 02:53:48.350275 412 raft_consensus.cc:2238] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:48.350387 412 raft_consensus.cc:2267] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:48.350836 412 tablet_replica.cc:331] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9: stopping tablet replica
I20250111 02:53:48.350945 412 raft_consensus.cc:2238] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:48.351051 412 raft_consensus.cc:2267] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:48.351516 412 tablet_replica.cc:331] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9: stopping tablet replica
I20250111 02:53:48.351622 412 raft_consensus.cc:2238] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:48.351724 412 raft_consensus.cc:2267] T 394a0b54dc6c458494b058a2c9547d20 P 1eeddf93653e4453847337013c1f19f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:48.365801 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
W20250111 02:53:48.654290 5951 consensus_peers.cc:487] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:48.672106 6111 consensus_queue.cc:1035] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:53:48.735327 5951 consensus_peers.cc:487] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:48.735780 6115 consensus_queue.cc:1035] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:53:48.740166 6025 consensus_peers.cc:487] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:48.837545 6111 consensus_queue.cc:1035] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:53:48.838426 5951 consensus_peers.cc:487] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:48.880477 6140 consensus_queue.cc:1035] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:53:49.369979 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:53:49.372718 6144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:49.373306 6145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:53:49.374444 6147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:53:49.375291 412 server_base.cc:1034] running on GCE node
I20250111 02:53:49.375634 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:53:49.375694 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:53:49.375725 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564029375720 us; error 0 us; skew 500 ppm
I20250111 02:53:49.375854 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:53:49.377080 412 webserver.cc:458] Webserver started at http://127.0.103.4:34207/ using document root <none> and password file <none>
I20250111 02:53:49.377311 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:53:49.377373 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:53:49.377506 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:53:49.378176 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-3-root/instance:
uuid: "1b31a733ff894169808ee120264a0202"
format_stamp: "Formatted at 2025-01-11 02:53:49 on dist-test-slave-4xrj"
I20250111 02:53:49.380631 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:49.382190 6152 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:49.382519 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250111 02:53:49.382615 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-3-root
uuid: "1b31a733ff894169808ee120264a0202"
format_stamp: "Formatted at 2025-01-11 02:53:49 on dist-test-slave-4xrj"
I20250111 02:53:49.382725 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1736563872491422-412-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:53:49.403564 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:53:49.404157 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:53:49.404695 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:53:49.405534 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:53:49.405592 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:49.405690 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:53:49.405732 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:53:49.414031 6140 consensus_queue.cc:579] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Leader has been unable to successfully communicate with peer 1eeddf93653e4453847337013c1f19f9 for more than 1 seconds (1.120s)
W20250111 02:53:49.417236 5809 catalog_manager.cc:5204] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ef1f1786674425a8ef7ad9a609eeac5 with cas_config_opid_index -1: no extra replica candidate found for tablet 8ef1f1786674425a8ef7ad9a609eeac5 (table test-workload [id=5001a8d5e9ac49538de667f4e02545e9]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250111 02:53:49.420272 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.4:37391
I20250111 02:53:49.420301 6216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.4:37391 every 8 connection(s)
I20250111 02:53:49.426383 6217 heartbeater.cc:346] Connected to a master server at 127.0.103.62:35083
I20250111 02:53:49.426512 6217 heartbeater.cc:463] Registering TS with master...
I20250111 02:53:49.426818 6217 heartbeater.cc:510] Master 127.0.103.62:35083 requested a full tablet report, sending...
I20250111 02:53:49.427445 5809 ts_manager.cc:194] Registered new tserver with Master: 1b31a733ff894169808ee120264a0202 (127.0.103.4:37391)
I20250111 02:53:49.428063 5809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48208
I20250111 02:53:49.473636 5986 consensus_queue.cc:237] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:49.476249 6060 raft_consensus.cc:1270] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:53:49.476994 6196 consensus_queue.cc:1035] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250111 02:53:49.477541 5951 consensus_peers.cc:487] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:49.478994 6140 raft_consensus.cc:2949] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
I20250111 02:53:49.479602 6060 raft_consensus.cc:2949] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
W20250111 02:53:49.482512 5952 consensus_peers.cc:487] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1b31a733ff894169808ee120264a0202 (127.0.103.4:37391): Couldn't send request to peer 1b31a733ff894169808ee120264a0202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ef1f1786674425a8ef7ad9a609eeac5. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:49.483255 5794 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8ef1f1786674425a8ef7ad9a609eeac5 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 2)
I20250111 02:53:49.484236 5809 catalog_manager.cc:5526] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c reported cstate change: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
I20250111 02:53:49.567894 6199 consensus_queue.cc:579] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Leader has been unable to successfully communicate with peer 1eeddf93653e4453847337013c1f19f9 for more than 1 seconds (1.307s)
I20250111 02:53:49.571331 5986 consensus_queue.cc:237] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:49.573100 6060 raft_consensus.cc:1270] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250111 02:53:49.573221 5952 consensus_peers.cc:487] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1b31a733ff894169808ee120264a0202 (127.0.103.4:37391): Couldn't send request to peer 1b31a733ff894169808ee120264a0202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 186ab318f3264d33ab17da2af5d96348. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:49.573552 6140 consensus_queue.cc:1035] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250111 02:53:49.573956 5951 consensus_peers.cc:487] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:49.575461 6199 raft_consensus.cc:2949] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
I20250111 02:53:49.575942 6060 raft_consensus.cc:2949] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
I20250111 02:53:49.578155 5794 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 186ab318f3264d33ab17da2af5d96348 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250111 02:53:49.579360 5807 catalog_manager.cc:5526] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:49.652081 6199 consensus_queue.cc:579] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Leader has been unable to successfully communicate with peer 1eeddf93653e4453847337013c1f19f9 for more than 1 seconds (1.406s)
I20250111 02:53:49.655433 5986 consensus_queue.cc:237] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:49.657204 6060 raft_consensus.cc:1270] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250111 02:53:49.657356 5952 consensus_peers.cc:487] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1b31a733ff894169808ee120264a0202 (127.0.103.4:37391): Couldn't send request to peer 1b31a733ff894169808ee120264a0202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 394a0b54dc6c458494b058a2c9547d20. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:49.657941 6199 consensus_queue.cc:1035] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250111 02:53:49.658123 5951 consensus_peers.cc:487] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:49.659652 6140 raft_consensus.cc:2949] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
I20250111 02:53:49.660166 6060 raft_consensus.cc:2949] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
I20250111 02:53:49.662433 5794 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 394a0b54dc6c458494b058a2c9547d20 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250111 02:53:49.663726 5809 catalog_manager.cc:5526] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:53:49.696318 6130 consensus_queue.cc:579] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Leader has been unable to successfully communicate with peer 1eeddf93653e4453847337013c1f19f9 for more than 1 seconds (1.420s)
I20250111 02:53:49.700074 6060 consensus_queue.cc:237] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:49.701937 5986 raft_consensus.cc:1270] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Refusing update from remote peer c067bca729e146388673a4278dbc5b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:53:49.702582 6228 consensus_queue.cc:1035] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250111 02:53:49.703783 6025 consensus_peers.cc:487] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:49.704654 6227 raft_consensus.cc:2949] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
I20250111 02:53:49.705230 5986 raft_consensus.cc:2949] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
W20250111 02:53:49.707942 6026 consensus_peers.cc:487] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c -> Peer 1b31a733ff894169808ee120264a0202 (127.0.103.4:37391): Couldn't send request to peer 1b31a733ff894169808ee120264a0202. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c15e3bbee23f43cd8b81c5d057ad7b32. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:49.708725 5795 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c15e3bbee23f43cd8b81c5d057ad7b32 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250111 02:53:49.710397 5807 catalog_manager.cc:5526] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: config changed from index -1 to 2, NON_VOTER 1b31a733ff894169808ee120264a0202 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "c067bca729e146388673a4278dbc5b4c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } } }
W20250111 02:53:49.822940 5857 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1eeddf93653e4453847337013c1f19f9 not available for placement
I20250111 02:53:50.041570 6231 ts_tablet_manager.cc:927] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: Initiating tablet copy from peer ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2:46259)
I20250111 02:53:50.042111 6231 tablet_copy_client.cc:323] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:46259
I20250111 02:53:50.051103 5996 tablet_copy_service.cc:140] P ee77e10db69e4f10a9bbae06ddbb112b: Received BeginTabletCopySession request for tablet 8ef1f1786674425a8ef7ad9a609eeac5 from peer 1b31a733ff894169808ee120264a0202 ({username='slave'} at 127.0.0.1:44550)
I20250111 02:53:50.051263 5996 tablet_copy_service.cc:161] P ee77e10db69e4f10a9bbae06ddbb112b: Beginning new tablet copy session on tablet 8ef1f1786674425a8ef7ad9a609eeac5 from peer 1b31a733ff894169808ee120264a0202 at {username='slave'} at 127.0.0.1:44550: session id = 1b31a733ff894169808ee120264a0202-8ef1f1786674425a8ef7ad9a609eeac5
I20250111 02:53:50.053308 5996 tablet_copy_source_session.cc:215] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:50.054416 6231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ef1f1786674425a8ef7ad9a609eeac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:50.058624 6231 tablet_copy_client.cc:806] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:50.058794 6231 tablet_copy_client.cc:670] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:50.060106 6231 tablet_copy_client.cc:538] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:50.062637 6231 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: Bootstrap starting.
I20250111 02:53:50.067140 6231 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:50.067428 6231 tablet_bootstrap.cc:492] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: Bootstrap complete.
I20250111 02:53:50.067623 6231 ts_tablet_manager.cc:1397] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250111 02:53:50.068202 6231 raft_consensus.cc:357] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:50.068361 6231 raft_consensus.cc:738] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b31a733ff894169808ee120264a0202, State: Initialized, Role: LEARNER
I20250111 02:53:50.068523 6231 consensus_queue.cc:260] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:50.069172 6231 ts_tablet_manager.cc:1428] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250111 02:53:50.069187 6217 heartbeater.cc:502] Master 127.0.103.62:35083 was elected leader, sending a full tablet report...
I20250111 02:53:50.069815 5996 tablet_copy_service.cc:342] P ee77e10db69e4f10a9bbae06ddbb112b: Request end of tablet copy session 1b31a733ff894169808ee120264a0202-8ef1f1786674425a8ef7ad9a609eeac5 received from {username='slave'} at 127.0.0.1:44550
I20250111 02:53:50.069980 5996 tablet_copy_service.cc:434] P ee77e10db69e4f10a9bbae06ddbb112b: ending tablet copy session 1b31a733ff894169808ee120264a0202-8ef1f1786674425a8ef7ad9a609eeac5 on tablet 8ef1f1786674425a8ef7ad9a609eeac5 with peer 1b31a733ff894169808ee120264a0202
W20250111 02:53:50.084455 5951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111) [suppressed 18 similar messages]
I20250111 02:53:50.094729 6231 ts_tablet_manager.cc:927] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: Initiating tablet copy from peer c067bca729e146388673a4278dbc5b4c (127.0.103.3:33649)
I20250111 02:53:50.095286 6231 tablet_copy_client.cc:323] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:33649
I20250111 02:53:50.100323 6070 tablet_copy_service.cc:140] P c067bca729e146388673a4278dbc5b4c: Received BeginTabletCopySession request for tablet c15e3bbee23f43cd8b81c5d057ad7b32 from peer 1b31a733ff894169808ee120264a0202 ({username='slave'} at 127.0.0.1:56186)
I20250111 02:53:50.100458 6070 tablet_copy_service.cc:161] P c067bca729e146388673a4278dbc5b4c: Beginning new tablet copy session on tablet c15e3bbee23f43cd8b81c5d057ad7b32 from peer 1b31a733ff894169808ee120264a0202 at {username='slave'} at 127.0.0.1:56186: session id = 1b31a733ff894169808ee120264a0202-c15e3bbee23f43cd8b81c5d057ad7b32
I20250111 02:53:50.102272 6070 tablet_copy_source_session.cc:215] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:50.103346 6231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c15e3bbee23f43cd8b81c5d057ad7b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:50.107329 6231 tablet_copy_client.cc:806] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:50.107510 6231 tablet_copy_client.cc:670] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:50.108848 6231 tablet_copy_client.cc:538] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:50.111042 6231 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: Bootstrap starting.
I20250111 02:53:50.115496 6240 ts_tablet_manager.cc:927] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: Initiating tablet copy from peer ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2:46259)
I20250111 02:53:50.116056 6231 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:50.116061 6240 tablet_copy_client.cc:323] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:46259
I20250111 02:53:50.116410 6231 tablet_bootstrap.cc:492] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: Bootstrap complete.
I20250111 02:53:50.116613 6231 ts_tablet_manager.cc:1397] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250111 02:53:50.116659 5996 tablet_copy_service.cc:140] P ee77e10db69e4f10a9bbae06ddbb112b: Received BeginTabletCopySession request for tablet 186ab318f3264d33ab17da2af5d96348 from peer 1b31a733ff894169808ee120264a0202 ({username='slave'} at 127.0.0.1:44550)
I20250111 02:53:50.116807 5996 tablet_copy_service.cc:161] P ee77e10db69e4f10a9bbae06ddbb112b: Beginning new tablet copy session on tablet 186ab318f3264d33ab17da2af5d96348 from peer 1b31a733ff894169808ee120264a0202 at {username='slave'} at 127.0.0.1:44550: session id = 1b31a733ff894169808ee120264a0202-186ab318f3264d33ab17da2af5d96348
I20250111 02:53:50.117267 6231 raft_consensus.cc:357] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:50.117466 6231 raft_consensus.cc:738] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b31a733ff894169808ee120264a0202, State: Initialized, Role: LEARNER
I20250111 02:53:50.117681 6231 consensus_queue.cc:260] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:50.118152 6231 ts_tablet_manager.cc:1428] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250111 02:53:50.118733 6070 tablet_copy_service.cc:342] P c067bca729e146388673a4278dbc5b4c: Request end of tablet copy session 1b31a733ff894169808ee120264a0202-c15e3bbee23f43cd8b81c5d057ad7b32 received from {username='slave'} at 127.0.0.1:56186
I20250111 02:53:50.118844 6070 tablet_copy_service.cc:434] P c067bca729e146388673a4278dbc5b4c: ending tablet copy session 1b31a733ff894169808ee120264a0202-c15e3bbee23f43cd8b81c5d057ad7b32 on tablet c15e3bbee23f43cd8b81c5d057ad7b32 with peer 1b31a733ff894169808ee120264a0202
I20250111 02:53:50.119216 5996 tablet_copy_source_session.cc:215] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:50.120198 6240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 186ab318f3264d33ab17da2af5d96348. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:50.124599 6240 tablet_copy_client.cc:806] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:50.124799 6240 tablet_copy_client.cc:670] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:50.125991 6240 tablet_copy_client.cc:538] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:50.128166 6240 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: Bootstrap starting.
I20250111 02:53:50.132699 6240 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:50.133046 6240 tablet_bootstrap.cc:492] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: Bootstrap complete.
I20250111 02:53:50.133262 6240 ts_tablet_manager.cc:1397] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250111 02:53:50.134090 6240 raft_consensus.cc:357] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:50.134274 6240 raft_consensus.cc:738] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b31a733ff894169808ee120264a0202, State: Initialized, Role: LEARNER
I20250111 02:53:50.134460 6240 consensus_queue.cc:260] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:50.135011 6240 ts_tablet_manager.cc:1428] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:53:50.135592 5996 tablet_copy_service.cc:342] P ee77e10db69e4f10a9bbae06ddbb112b: Request end of tablet copy session 1b31a733ff894169808ee120264a0202-186ab318f3264d33ab17da2af5d96348 received from {username='slave'} at 127.0.0.1:44550
I20250111 02:53:50.135712 5996 tablet_copy_service.cc:434] P ee77e10db69e4f10a9bbae06ddbb112b: ending tablet copy session 1b31a733ff894169808ee120264a0202-186ab318f3264d33ab17da2af5d96348 on tablet 186ab318f3264d33ab17da2af5d96348 with peer 1b31a733ff894169808ee120264a0202
I20250111 02:53:50.212260 6240 ts_tablet_manager.cc:927] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: Initiating tablet copy from peer ee77e10db69e4f10a9bbae06ddbb112b (127.0.103.2:46259)
I20250111 02:53:50.212939 6240 tablet_copy_client.cc:323] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.2:46259
I20250111 02:53:50.213534 5996 tablet_copy_service.cc:140] P ee77e10db69e4f10a9bbae06ddbb112b: Received BeginTabletCopySession request for tablet 394a0b54dc6c458494b058a2c9547d20 from peer 1b31a733ff894169808ee120264a0202 ({username='slave'} at 127.0.0.1:44550)
I20250111 02:53:50.213661 5996 tablet_copy_service.cc:161] P ee77e10db69e4f10a9bbae06ddbb112b: Beginning new tablet copy session on tablet 394a0b54dc6c458494b058a2c9547d20 from peer 1b31a733ff894169808ee120264a0202 at {username='slave'} at 127.0.0.1:44550: session id = 1b31a733ff894169808ee120264a0202-394a0b54dc6c458494b058a2c9547d20
I20250111 02:53:50.215459 5996 tablet_copy_source_session.cc:215] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:53:50.216473 6240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 394a0b54dc6c458494b058a2c9547d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:53:50.220772 6240 tablet_copy_client.cc:806] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: tablet copy: Starting download of 0 data blocks...
I20250111 02:53:50.220948 6240 tablet_copy_client.cc:670] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: tablet copy: Starting download of 1 WAL segments...
I20250111 02:53:50.222263 6240 tablet_copy_client.cc:538] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:53:50.225085 6240 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: Bootstrap starting.
I20250111 02:53:50.230224 6240 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:53:50.230532 6240 tablet_bootstrap.cc:492] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: Bootstrap complete.
I20250111 02:53:50.230742 6240 ts_tablet_manager.cc:1397] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20250111 02:53:50.231323 6240 raft_consensus.cc:357] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:50.231510 6240 raft_consensus.cc:738] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1b31a733ff894169808ee120264a0202, State: Initialized, Role: LEARNER
I20250111 02:53:50.231695 6240 consensus_queue.cc:260] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: true } }
I20250111 02:53:50.232263 6240 ts_tablet_manager.cc:1428] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:53:50.232949 5996 tablet_copy_service.cc:342] P ee77e10db69e4f10a9bbae06ddbb112b: Request end of tablet copy session 1b31a733ff894169808ee120264a0202-394a0b54dc6c458494b058a2c9547d20 received from {username='slave'} at 127.0.0.1:44550
I20250111 02:53:50.233079 5996 tablet_copy_service.cc:434] P ee77e10db69e4f10a9bbae06ddbb112b: ending tablet copy session 1b31a733ff894169808ee120264a0202-394a0b54dc6c458494b058a2c9547d20 on tablet 394a0b54dc6c458494b058a2c9547d20 with peer 1b31a733ff894169808ee120264a0202
I20250111 02:53:50.471571 6190 raft_consensus.cc:1212] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250111 02:53:50.536680 6190 raft_consensus.cc:1212] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250111 02:53:50.601554 6190 raft_consensus.cc:1212] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250111 02:53:50.650903 6245 consensus_queue.cc:579] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Leader has been unable to successfully communicate with peer 1eeddf93653e4453847337013c1f19f9 for more than 1 seconds (1.177s)
I20250111 02:53:50.693222 6245 consensus_queue.cc:579] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Leader has been unable to successfully communicate with peer 1eeddf93653e4453847337013c1f19f9 for more than 1 seconds (1.121s)
I20250111 02:53:50.712515 6248 consensus_queue.cc:579] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Leader has been unable to successfully communicate with peer 1eeddf93653e4453847337013c1f19f9 for more than 1 seconds (1.012s)
I20250111 02:53:50.779302 6190 raft_consensus.cc:1212] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250111 02:53:50.823742 5857 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 1eeddf93653e4453847337013c1f19f9 not available for placement
I20250111 02:53:50.879871 6245 raft_consensus.cc:1059] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b: attempting to promote NON_VOTER 1b31a733ff894169808ee120264a0202 to VOTER
I20250111 02:53:50.880513 6245 consensus_queue.cc:237] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } }
I20250111 02:53:50.882532 6190 raft_consensus.cc:1270] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:50.882532 6060 raft_consensus.cc:1270] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:50.883102 6254 consensus_queue.cc:1035] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:53:50.883366 6245 consensus_queue.cc:1035] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250111 02:53:50.884006 5951 consensus_peers.cc:487] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:50.886207 6254 raft_consensus.cc:2949] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.886902 6190 raft_consensus.cc:2949] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.889473 6060 raft_consensus.cc:2949] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.890331 5809 catalog_manager.cc:5526] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:50.895587 5986 consensus_queue.cc:237] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } }
I20250111 02:53:50.897099 6190 raft_consensus.cc:1270] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:50.897279 6060 raft_consensus.cc:1270] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:50.897536 6246 consensus_queue.cc:1035] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:50.897811 6245 consensus_queue.cc:1035] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:50.899482 6246 raft_consensus.cc:2949] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.900020 6190 raft_consensus.cc:2949] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.900298 6060 raft_consensus.cc:2949] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.902213 5794 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 8ef1f1786674425a8ef7ad9a609eeac5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:50.903568 5808 catalog_manager.cc:5526] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250111 02:53:50.909026 5795 catalog_manager.cc:4615] TS 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8ef1f1786674425a8ef7ad9a609eeac5: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111)
I20250111 02:53:50.949942 6254 raft_consensus.cc:1059] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b: attempting to promote NON_VOTER 1b31a733ff894169808ee120264a0202 to VOTER
I20250111 02:53:50.950628 6254 consensus_queue.cc:237] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } }
I20250111 02:53:50.952661 6060 raft_consensus.cc:1270] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:50.952672 6190 raft_consensus.cc:1270] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:50.953128 6246 consensus_queue.cc:1035] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:53:50.953367 6245 consensus_queue.cc:1035] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250111 02:53:50.953610 5951 consensus_peers.cc:487] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:50.956179 6246 raft_consensus.cc:2949] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.956725 6190 raft_consensus.cc:2949] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.959415 6060 raft_consensus.cc:2949] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.960438 5809 catalog_manager.cc:5526] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 reported cstate change: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.965384 5986 consensus_queue.cc:237] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } }
I20250111 02:53:50.966857 6190 raft_consensus.cc:1270] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:50.966919 6060 raft_consensus.cc:1270] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:50.967299 6246 consensus_queue.cc:1035] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:50.967484 6245 consensus_queue.cc:1035] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:50.969319 6196 raft_consensus.cc:2949] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.969863 6060 raft_consensus.cc:2949] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.972290 5794 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 186ab318f3264d33ab17da2af5d96348 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:50.972719 6190 raft_consensus.cc:2949] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:50.973001 5808 catalog_manager.cc:5526] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:53:51.040036 6228 raft_consensus.cc:1059] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c: attempting to promote NON_VOTER 1b31a733ff894169808ee120264a0202 to VOTER
I20250111 02:53:51.040670 6228 consensus_queue.cc:237] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } }
I20250111 02:53:51.042727 5986 raft_consensus.cc:1270] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Refusing update from remote peer c067bca729e146388673a4278dbc5b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:51.043066 6190 raft_consensus.cc:1270] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Refusing update from remote peer c067bca729e146388673a4278dbc5b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:51.043241 6247 consensus_queue.cc:1035] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:53:51.043596 6267 consensus_queue.cc:1035] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250111 02:53:51.043839 6025 consensus_peers.cc:487] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:51.046015 6247 raft_consensus.cc:2949] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.046558 6190 raft_consensus.cc:2949] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.049831 5986 raft_consensus.cc:2949] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.050693 5808 catalog_manager.cc:5526] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 reported cstate change: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c067bca729e146388673a4278dbc5b4c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.054811 6060 consensus_queue.cc:237] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } }
I20250111 02:53:51.056478 6190 raft_consensus.cc:1270] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Refusing update from remote peer c067bca729e146388673a4278dbc5b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:51.056710 5986 raft_consensus.cc:1270] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Refusing update from remote peer c067bca729e146388673a4278dbc5b4c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:51.057060 6247 consensus_queue.cc:1035] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:51.057566 6267 consensus_queue.cc:1035] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:51.059371 6248 raft_consensus.cc:2949] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.060083 6190 raft_consensus.cc:2949] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.060097 5986 raft_consensus.cc:2949] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.064087 5795 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet c15e3bbee23f43cd8b81c5d057ad7b32 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:51.064019 5807 catalog_manager.cc:5526] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 reported cstate change: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New cstate: current_term: 1 leader_uuid: "c067bca729e146388673a4278dbc5b4c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.243191 6254 consensus_queue.cc:579] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Leader has been unable to successfully communicate with peer 1eeddf93653e4453847337013c1f19f9 for more than 1 seconds (1.587s)
I20250111 02:53:51.280943 6245 raft_consensus.cc:1059] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b: attempting to promote NON_VOTER 1b31a733ff894169808ee120264a0202 to VOTER
I20250111 02:53:51.281710 6245 consensus_queue.cc:237] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } }
I20250111 02:53:51.284904 6060 raft_consensus.cc:1270] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:51.285435 6245 consensus_queue.cc:1035] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250111 02:53:51.286001 5951 consensus_peers.cc:487] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b -> Peer 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): Couldn't send request to peer 1eeddf93653e4453847337013c1f19f9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:53:51.286095 6190 raft_consensus.cc:1270] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 LEARNER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:53:51.286566 6245 consensus_queue.cc:1035] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:53:51.288899 6246 raft_consensus.cc:2949] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.289533 6190 raft_consensus.cc:2949] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.293249 6060 raft_consensus.cc:2949] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.294201 5807 catalog_manager.cc:5526] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 reported cstate change: config changed from index 2 to 3, 1b31a733ff894169808ee120264a0202 (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1eeddf93653e4453847337013c1f19f9" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 39533 } } peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.298532 5986 consensus_queue.cc:237] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } }
I20250111 02:53:51.299976 6190 raft_consensus.cc:1270] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:51.300050 6060 raft_consensus.cc:1270] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Refusing update from remote peer ee77e10db69e4f10a9bbae06ddbb112b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:53:51.300412 6246 consensus_queue.cc:1035] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:51.300621 6245 consensus_queue.cc:1035] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:53:51.302520 6254 raft_consensus.cc:2949] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.303076 6060 raft_consensus.cc:2949] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.303076 6190 raft_consensus.cc:2949] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } } }
I20250111 02:53:51.305552 5794 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 394a0b54dc6c458494b058a2c9547d20 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:53:51.306674 5807 catalog_manager.cc:5526] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b reported cstate change: config changed from index 3 to 4, VOTER 1eeddf93653e4453847337013c1f19f9 (127.0.103.1) evicted. New cstate: current_term: 1 leader_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ee77e10db69e4f10a9bbae06ddbb112b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c067bca729e146388673a4278dbc5b4c" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 33649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b31a733ff894169808ee120264a0202" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 37391 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250111 02:53:51.944583 5795 catalog_manager.cc:4615] TS 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 394a0b54dc6c458494b058a2c9547d20: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111)
W20250111 02:53:53.080785 5795 catalog_manager.cc:4615] TS 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8ef1f1786674425a8ef7ad9a609eeac5: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111)
W20250111 02:53:55.164979 5795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111) [suppressed 44 similar messages]
W20250111 02:53:55.166154 5795 catalog_manager.cc:4615] TS 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8ef1f1786674425a8ef7ad9a609eeac5: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111)
I20250111 02:53:56.459571 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:53:56.467558 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:56.467818 412 tablet_replica.cc:331] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b: stopping tablet replica
I20250111 02:53:56.467995 412 raft_consensus.cc:2238] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:56.468299 412 raft_consensus.cc:2267] T 186ab318f3264d33ab17da2af5d96348 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.468869 412 tablet_replica.cc:331] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b: stopping tablet replica
I20250111 02:53:56.469004 412 raft_consensus.cc:2238] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:56.469216 412 raft_consensus.cc:2267] T c15e3bbee23f43cd8b81c5d057ad7b32 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.469621 412 tablet_replica.cc:331] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b: stopping tablet replica
I20250111 02:53:56.469745 412 raft_consensus.cc:2238] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:56.470001 412 raft_consensus.cc:2267] T 394a0b54dc6c458494b058a2c9547d20 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.470386 412 tablet_replica.cc:331] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b: stopping tablet replica
I20250111 02:53:56.470510 412 raft_consensus.cc:2238] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:56.470746 412 raft_consensus.cc:2267] T 8ef1f1786674425a8ef7ad9a609eeac5 P ee77e10db69e4f10a9bbae06ddbb112b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.484735 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:53:56.488107 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:53:56.496829 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:56.497112 412 tablet_replica.cc:331] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c: stopping tablet replica
I20250111 02:53:56.497282 412 raft_consensus.cc:2238] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:53:56.497583 412 raft_consensus.cc:2267] T c15e3bbee23f43cd8b81c5d057ad7b32 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.498147 412 tablet_replica.cc:331] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c: stopping tablet replica
I20250111 02:53:56.498286 412 raft_consensus.cc:2238] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:56.498497 412 raft_consensus.cc:2267] T 186ab318f3264d33ab17da2af5d96348 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.498914 412 tablet_replica.cc:331] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c: stopping tablet replica
I20250111 02:53:56.499024 412 raft_consensus.cc:2238] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:56.499204 412 raft_consensus.cc:2267] T 8ef1f1786674425a8ef7ad9a609eeac5 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.499590 412 tablet_replica.cc:331] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c: stopping tablet replica
I20250111 02:53:56.499701 412 raft_consensus.cc:2238] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:56.499871 412 raft_consensus.cc:2267] T 394a0b54dc6c458494b058a2c9547d20 P c067bca729e146388673a4278dbc5b4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.513883 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:53:56.517272 412 tablet_server.cc:178] TabletServer@127.0.103.4:0 shutting down...
I20250111 02:53:56.526696 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:53:56.526934 412 tablet_replica.cc:331] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202: stopping tablet replica
I20250111 02:53:56.527137 412 raft_consensus.cc:2238] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:56.527374 412 raft_consensus.cc:2267] T 394a0b54dc6c458494b058a2c9547d20 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.527930 412 tablet_replica.cc:331] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202: stopping tablet replica
I20250111 02:53:56.528044 412 raft_consensus.cc:2238] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:56.528196 412 raft_consensus.cc:2267] T 186ab318f3264d33ab17da2af5d96348 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.528558 412 tablet_replica.cc:331] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202: stopping tablet replica
I20250111 02:53:56.528649 412 raft_consensus.cc:2238] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:56.528796 412 raft_consensus.cc:2267] T 8ef1f1786674425a8ef7ad9a609eeac5 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.529130 412 tablet_replica.cc:331] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202: stopping tablet replica
I20250111 02:53:56.529215 412 raft_consensus.cc:2238] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:53:56.529347 412 raft_consensus.cc:2267] T c15e3bbee23f43cd8b81c5d057ad7b32 P 1b31a733ff894169808ee120264a0202 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:53:56.543850 412 tablet_server.cc:195] TabletServer@127.0.103.4:0 shutdown complete.
I20250111 02:53:56.547272 412 master.cc:537] Master@127.0.103.62:35083 shutting down...
W20250111 02:53:59.275660 5795 catalog_manager.cc:4615] TS 1eeddf93653e4453847337013c1f19f9 (127.0.103.1:39533): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8ef1f1786674425a8ef7ad9a609eeac5: Network error: Client connection negotiation failed: client connection to 127.0.103.1:39533: connect: Connection refused (error 111)
I20250111 02:54:00.157961 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:00.158262 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:00.158351 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 47efffa89b6a4a41bf34d8161dd1cb77: stopping tablet replica
I20250111 02:54:00.186903 412 master.cc:559] Master@127.0.103.62:35083 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13432 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20250111 02:54:00.201503 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:38351
I20250111 02:54:00.202046 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:00.204568 6279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:00.205291 6280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:00.206288 6282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:00.207165 412 server_base.cc:1034] running on GCE node
I20250111 02:54:00.207377 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:00.207445 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:00.207501 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564040207484 us; error 0 us; skew 500 ppm
I20250111 02:54:00.207664 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:00.208930 412 webserver.cc:458] Webserver started at http://127.0.103.62:36401/ using document root <none> and password file <none>
I20250111 02:54:00.209170 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:00.209254 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:00.209403 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:00.210096 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "375dab4837a34932b9d2fcb074d6b7b6"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:00.212616 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:00.214257 6287 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.214677 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250111 02:54:00.214777 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "375dab4837a34932b9d2fcb074d6b7b6"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:00.214890 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:00.231133 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:00.231652 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:00.241782 6338 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:38351 every 8 connection(s)
I20250111 02:54:00.241761 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:38351
I20250111 02:54:00.243044 6339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:00.247028 6339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6: Bootstrap starting.
I20250111 02:54:00.248152 6339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:00.249504 6339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6: No bootstrap required, opened a new log
I20250111 02:54:00.250098 6339 raft_consensus.cc:357] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375dab4837a34932b9d2fcb074d6b7b6" member_type: VOTER }
I20250111 02:54:00.250226 6339 raft_consensus.cc:383] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:00.250272 6339 raft_consensus.cc:738] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 375dab4837a34932b9d2fcb074d6b7b6, State: Initialized, Role: FOLLOWER
I20250111 02:54:00.250504 6339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375dab4837a34932b9d2fcb074d6b7b6" member_type: VOTER }
I20250111 02:54:00.250644 6339 raft_consensus.cc:397] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:54:00.250734 6339 raft_consensus.cc:491] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:54:00.250823 6339 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:00.252405 6339 raft_consensus.cc:513] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375dab4837a34932b9d2fcb074d6b7b6" member_type: VOTER }
I20250111 02:54:00.252593 6339 leader_election.cc:304] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 375dab4837a34932b9d2fcb074d6b7b6; no voters:
I20250111 02:54:00.252974 6339 leader_election.cc:290] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:54:00.253085 6342 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:00.253538 6342 raft_consensus.cc:695] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 1 LEADER]: Becoming Leader. State: Replica: 375dab4837a34932b9d2fcb074d6b7b6, State: Running, Role: LEADER
I20250111 02:54:00.253834 6339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:54:00.253794 6342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375dab4837a34932b9d2fcb074d6b7b6" member_type: VOTER }
I20250111 02:54:00.255088 6343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "375dab4837a34932b9d2fcb074d6b7b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375dab4837a34932b9d2fcb074d6b7b6" member_type: VOTER } }
I20250111 02:54:00.255137 6344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 375dab4837a34932b9d2fcb074d6b7b6. Latest consensus state: current_term: 1 leader_uuid: "375dab4837a34932b9d2fcb074d6b7b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "375dab4837a34932b9d2fcb074d6b7b6" member_type: VOTER } }
I20250111 02:54:00.255293 6343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [sys.catalog]: This master's current role is: LEADER
I20250111 02:54:00.255298 6344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [sys.catalog]: This master's current role is: LEADER
I20250111 02:54:00.256258 6347 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:54:00.257710 6347 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:54:00.259621 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:54:00.260573 6347 catalog_manager.cc:1348] Generated new cluster ID: a99514941c5c4aeeb70426cb4c0d780a
I20250111 02:54:00.260670 6347 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:54:00.298760 6347 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:54:00.299511 6347 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:54:00.306402 6347 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6: Generated new TSK 0
I20250111 02:54:00.306658 6347 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:54:00.324584 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:00.327390 6360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:00.327873 6361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:00.328832 6363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:00.329506 412 server_base.cc:1034] running on GCE node
I20250111 02:54:00.329737 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:00.329797 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:00.329839 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564040329825 us; error 0 us; skew 500 ppm
I20250111 02:54:00.330008 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:00.331316 412 webserver.cc:458] Webserver started at http://127.0.103.1:39569/ using document root <none> and password file <none>
I20250111 02:54:00.331538 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:00.331606 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:00.331735 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:00.332381 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "bd3f4281bc0f4958b77f4304323bbd07"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:00.334915 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:54:00.336429 6368 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.336781 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:54:00.336892 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "bd3f4281bc0f4958b77f4304323bbd07"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:00.337005 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:00.355808 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:00.356346 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:00.356909 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:00.357836 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:00.357957 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.358037 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:00.358083 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.369414 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:46417
I20250111 02:54:00.369441 6430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:46417 every 8 connection(s)
I20250111 02:54:00.371165 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:00.374231 6435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:00.374873 6436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:00.378190 6431 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38351
I20250111 02:54:00.378345 412 server_base.cc:1034] running on GCE node
I20250111 02:54:00.378346 6431 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:00.378803 6431 heartbeater.cc:510] Master 127.0.103.62:38351 requested a full tablet report, sending...
W20250111 02:54:00.378844 6438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:00.379258 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:00.379355 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:00.379420 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564040379396 us; error 0 us; skew 500 ppm
I20250111 02:54:00.379598 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:00.379803 6304 ts_manager.cc:194] Registered new tserver with Master: bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417)
I20250111 02:54:00.381036 412 webserver.cc:458] Webserver started at http://127.0.103.2:46045/ using document root <none> and password file <none>
I20250111 02:54:00.381294 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:00.381393 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:00.381371 6304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39228
I20250111 02:54:00.381534 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:00.382237 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "f920acdfe48e48cb9bd1463983b819f1"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:00.384601 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:00.386090 6443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.386413 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250111 02:54:00.386533 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "f920acdfe48e48cb9bd1463983b819f1"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:00.386647 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:00.412135 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:00.412791 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:00.413341 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:00.414160 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:00.414237 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.414338 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:00.414402 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.425680 6505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:43471 every 8 connection(s)
I20250111 02:54:00.425655 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:43471
I20250111 02:54:00.427258 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:00.430377 6509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:00.431167 6510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:00.431564 6512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:00.432709 412 server_base.cc:1034] running on GCE node
I20250111 02:54:00.432998 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:00.433117 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:00.433192 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564040433166 us; error 0 us; skew 500 ppm
I20250111 02:54:00.433382 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:00.434379 6506 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38351
I20250111 02:54:00.434520 6506 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:00.434919 6506 heartbeater.cc:510] Master 127.0.103.62:38351 requested a full tablet report, sending...
I20250111 02:54:00.434948 412 webserver.cc:458] Webserver started at http://127.0.103.3:33311/ using document root <none> and password file <none>
I20250111 02:54:00.435212 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:00.435317 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:00.435475 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:00.435920 6304 ts_manager.cc:194] Registered new tserver with Master: f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471)
I20250111 02:54:00.436271 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "16ab852a34d94de69641565f2bbd2881"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:00.436962 6304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39236
I20250111 02:54:00.439226 412 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250111 02:54:00.440717 6517 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.441046 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:54:00.441154 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "16ab852a34d94de69641565f2bbd2881"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:00.441268 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:00.451140 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:00.451716 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:00.452225 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:00.453020 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:00.453104 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.453204 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:00.453265 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:00.464938 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:40961
I20250111 02:54:00.464962 6579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:40961 every 8 connection(s)
I20250111 02:54:00.471535 6580 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38351
I20250111 02:54:00.471688 6580 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:00.472018 6580 heartbeater.cc:510] Master 127.0.103.62:38351 requested a full tablet report, sending...
I20250111 02:54:00.472826 6304 ts_manager.cc:194] Registered new tserver with Master: 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:00.473541 6304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39250
I20250111 02:54:00.473755 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007910404s
I20250111 02:54:01.382758 6431 heartbeater.cc:502] Master 127.0.103.62:38351 was elected leader, sending a full tablet report...
I20250111 02:54:01.438463 6506 heartbeater.cc:502] Master 127.0.103.62:38351 was elected leader, sending a full tablet report...
I20250111 02:54:01.474936 6580 heartbeater.cc:502] Master 127.0.103.62:38351 was elected leader, sending a full tablet report...
I20250111 02:54:01.508013 412 test_util.cc:274] Using random seed: 1389552794
I20250111 02:54:01.519698 6304 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39256:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:54:01.520434 6304 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:54:01.551505 6542 tablet_service.cc:1467] Processing CreateTablet for tablet 0741856ace474541b370b5579ab9093b (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:01.551841 6544 tablet_service.cc:1467] Processing CreateTablet for tablet d9c4f53f3a2e4b49b9ef215ea5454686 (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:01.552088 6542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0741856ace474541b370b5579ab9093b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.552496 6543 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4cf57ca748467ead76333d1e58a20f (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:01.552914 6543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4cf57ca748467ead76333d1e58a20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.554212 6471 tablet_service.cc:1467] Processing CreateTablet for tablet 486aa5274c4b4f3aa94a5c82682d146f (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:01.554323 6469 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4cf57ca748467ead76333d1e58a20f (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:01.554680 6471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486aa5274c4b4f3aa94a5c82682d146f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.554919 6469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4cf57ca748467ead76333d1e58a20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.555097 6395 tablet_service.cc:1467] Processing CreateTablet for tablet d9c4f53f3a2e4b49b9ef215ea5454686 (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:01.555512 6395 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9c4f53f3a2e4b49b9ef215ea5454686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.557317 6394 tablet_service.cc:1467] Processing CreateTablet for tablet 3f4cf57ca748467ead76333d1e58a20f (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:01.557695 6394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f4cf57ca748467ead76333d1e58a20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.558218 6470 tablet_service.cc:1467] Processing CreateTablet for tablet d9c4f53f3a2e4b49b9ef215ea5454686 (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:01.558678 6470 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9c4f53f3a2e4b49b9ef215ea5454686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.561528 6393 tablet_service.cc:1467] Processing CreateTablet for tablet 0741856ace474541b370b5579ab9093b (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:01.562140 6393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0741856ace474541b370b5579ab9093b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.562222 6468 tablet_service.cc:1467] Processing CreateTablet for tablet 0741856ace474541b370b5579ab9093b (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:01.562642 6468 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0741856ace474541b370b5579ab9093b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.554702 6396 tablet_service.cc:1467] Processing CreateTablet for tablet 486aa5274c4b4f3aa94a5c82682d146f (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:01.566532 6396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486aa5274c4b4f3aa94a5c82682d146f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.551505 6545 tablet_service.cc:1467] Processing CreateTablet for tablet 486aa5274c4b4f3aa94a5c82682d146f (DEFAULT_TABLE table=test-workload [id=ba1589a178e144159622de9846abf9ed]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:01.570643 6545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486aa5274c4b4f3aa94a5c82682d146f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.572402 6601 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap starting.
I20250111 02:54:01.574119 6544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9c4f53f3a2e4b49b9ef215ea5454686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:01.574182 6601 tablet_bootstrap.cc:654] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.576339 6601 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: No bootstrap required, opened a new log
I20250111 02:54:01.576490 6601 ts_tablet_manager.cc:1397] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:54:01.577296 6601 raft_consensus.cc:357] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.577523 6601 raft_consensus.cc:383] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.577603 6601 raft_consensus.cc:738] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.577904 6601 consensus_queue.cc:260] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.620057 6600 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Bootstrap starting.
I20250111 02:54:01.621070 6604 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Bootstrap starting.
I20250111 02:54:01.621472 6601 ts_tablet_manager.cc:1428] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Time spent starting tablet: real 0.045s user 0.016s sys 0.027s
W20250111 02:54:01.621893 6432 tablet.cc:2367] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250111 02:54:01.622475 6601 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap starting.
I20250111 02:54:01.622534 6600 tablet_bootstrap.cc:654] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.623076 6604 tablet_bootstrap.cc:654] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.624223 6601 tablet_bootstrap.cc:654] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.627737 6600 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: No bootstrap required, opened a new log
I20250111 02:54:01.627897 6600 ts_tablet_manager.cc:1397] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250111 02:54:01.628116 6604 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: No bootstrap required, opened a new log
I20250111 02:54:01.628158 6601 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: No bootstrap required, opened a new log
I20250111 02:54:01.628278 6604 ts_tablet_manager.cc:1397] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250111 02:54:01.628283 6601 ts_tablet_manager.cc:1397] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20250111 02:54:01.628720 6600 raft_consensus.cc:357] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.628911 6600 raft_consensus.cc:383] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.629002 6600 raft_consensus.cc:738] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.629055 6604 raft_consensus.cc:357] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.629071 6601 raft_consensus.cc:357] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.629324 6604 raft_consensus.cc:383] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.629354 6601 raft_consensus.cc:383] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.629261 6600 consensus_queue.cc:260] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.629454 6604 raft_consensus.cc:738] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.629457 6601 raft_consensus.cc:738] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.629742 6601 consensus_queue.cc:260] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.629724 6604 consensus_queue.cc:260] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.630219 6600 ts_tablet_manager.cc:1428] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250111 02:54:01.630347 6601 ts_tablet_manager.cc:1428] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250111 02:54:01.630695 6601 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap starting.
I20250111 02:54:01.630769 6604 ts_tablet_manager.cc:1428] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250111 02:54:01.631047 6604 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Bootstrap starting.
I20250111 02:54:01.632388 6601 tablet_bootstrap.cc:654] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.632388 6604 tablet_bootstrap.cc:654] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.634311 6600 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Bootstrap starting.
I20250111 02:54:01.634287 6601 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: No bootstrap required, opened a new log
I20250111 02:54:01.634531 6601 ts_tablet_manager.cc:1397] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20250111 02:54:01.635231 6601 raft_consensus.cc:357] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.636502 6601 raft_consensus.cc:383] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.636593 6601 raft_consensus.cc:738] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.636050 6600 tablet_bootstrap.cc:654] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.636874 6601 consensus_queue.cc:260] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.637645 6601 ts_tablet_manager.cc:1428] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250111 02:54:01.637976 6601 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap starting.
I20250111 02:54:01.638656 6604 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: No bootstrap required, opened a new log
I20250111 02:54:01.638855 6604 ts_tablet_manager.cc:1397] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20250111 02:54:01.639709 6601 tablet_bootstrap.cc:654] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.639699 6604 raft_consensus.cc:357] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.639945 6604 raft_consensus.cc:383] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.640072 6604 raft_consensus.cc:738] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.640357 6604 consensus_queue.cc:260] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.641036 6604 ts_tablet_manager.cc:1428] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250111 02:54:01.641418 6604 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Bootstrap starting.
I20250111 02:54:01.643137 6600 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: No bootstrap required, opened a new log
I20250111 02:54:01.643349 6600 ts_tablet_manager.cc:1397] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20250111 02:54:01.643520 6604 tablet_bootstrap.cc:654] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.644328 6600 raft_consensus.cc:357] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.644682 6600 raft_consensus.cc:383] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.644794 6600 raft_consensus.cc:738] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.645058 6600 consensus_queue.cc:260] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.645542 6604 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: No bootstrap required, opened a new log
I20250111 02:54:01.645607 6600 ts_tablet_manager.cc:1428] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250111 02:54:01.645742 6604 ts_tablet_manager.cc:1397] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250111 02:54:01.646221 6600 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Bootstrap starting.
I20250111 02:54:01.646574 6604 raft_consensus.cc:357] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.646836 6604 raft_consensus.cc:383] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.646953 6604 raft_consensus.cc:738] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.647295 6604 consensus_queue.cc:260] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.647900 6604 ts_tablet_manager.cc:1428] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:01.648195 6604 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Bootstrap starting.
I20250111 02:54:01.649801 6604 tablet_bootstrap.cc:654] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.651453 6600 tablet_bootstrap.cc:654] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.652666 6604 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: No bootstrap required, opened a new log
I20250111 02:54:01.652818 6604 ts_tablet_manager.cc:1397] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250111 02:54:01.653266 6601 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: No bootstrap required, opened a new log
I20250111 02:54:01.653431 6601 ts_tablet_manager.cc:1397] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20250111 02:54:01.653496 6600 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: No bootstrap required, opened a new log
I20250111 02:54:01.653674 6600 ts_tablet_manager.cc:1397] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20250111 02:54:01.653607 6604 raft_consensus.cc:357] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.653874 6604 raft_consensus.cc:383] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.653986 6604 raft_consensus.cc:738] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.654225 6604 consensus_queue.cc:260] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.654261 6601 raft_consensus.cc:357] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.654502 6601 raft_consensus.cc:383] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.654397 6600 raft_consensus.cc:357] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.654626 6601 raft_consensus.cc:738] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.654675 6600 raft_consensus.cc:383] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.654755 6604 ts_tablet_manager.cc:1428] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:01.654834 6600 raft_consensus.cc:738] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.654979 6601 consensus_queue.cc:260] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.655098 6600 consensus_queue.cc:260] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.655617 6600 ts_tablet_manager.cc:1428] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250111 02:54:01.655969 6600 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Bootstrap starting.
I20250111 02:54:01.656682 6601 ts_tablet_manager.cc:1428] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:54:01.657392 6600 tablet_bootstrap.cc:654] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:01.660671 6600 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: No bootstrap required, opened a new log
I20250111 02:54:01.660801 6600 ts_tablet_manager.cc:1397] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250111 02:54:01.661345 6600 raft_consensus.cc:357] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.661509 6600 raft_consensus.cc:383] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:01.661581 6600 raft_consensus.cc:738] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Initialized, Role: FOLLOWER
I20250111 02:54:01.661790 6600 consensus_queue.cc:260] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.662261 6600 ts_tablet_manager.cc:1428] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250111 02:54:01.735788 6609 raft_consensus.cc:491] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:01.735996 6609 raft_consensus.cc:513] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.736886 6609 leader_election.cc:290] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417)
I20250111 02:54:01.742858 6406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486aa5274c4b4f3aa94a5c82682d146f" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07" is_pre_election: true
I20250111 02:54:01.742858 6481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486aa5274c4b4f3aa94a5c82682d146f" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1" is_pre_election: true
I20250111 02:54:01.743206 6481 raft_consensus.cc:2463] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 0.
I20250111 02:54:01.743206 6406 raft_consensus.cc:2463] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 0.
I20250111 02:54:01.743690 6518 leader_election.cc:304] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:01.743988 6609 raft_consensus.cc:2798] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:01.744094 6609 raft_consensus.cc:491] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:01.744175 6609 raft_consensus.cc:3054] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.746057 6609 raft_consensus.cc:513] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.746640 6609 leader_election.cc:290] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 1 election: Requested vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417)
I20250111 02:54:01.747144 6481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486aa5274c4b4f3aa94a5c82682d146f" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1"
I20250111 02:54:01.747150 6406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486aa5274c4b4f3aa94a5c82682d146f" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07"
I20250111 02:54:01.747363 6481 raft_consensus.cc:3054] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.747411 6406 raft_consensus.cc:3054] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.749465 6481 raft_consensus.cc:2463] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 1.
I20250111 02:54:01.749477 6406 raft_consensus.cc:2463] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 1.
I20250111 02:54:01.749927 6518 leader_election.cc:304] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:01.750181 6609 raft_consensus.cc:2798] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:01.750614 6609 raft_consensus.cc:695] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 LEADER]: Becoming Leader. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Running, Role: LEADER
I20250111 02:54:01.750869 6609 consensus_queue.cc:237] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:01.753623 6304 catalog_manager.cc:5526] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16ab852a34d94de69641565f2bbd2881 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "16ab852a34d94de69641565f2bbd2881" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:01.802958 6609 raft_consensus.cc:491] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:01.803184 6609 raft_consensus.cc:513] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.803853 6609 leader_election.cc:290] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471)
I20250111 02:54:01.804354 6406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07" is_pre_election: true
I20250111 02:54:01.804414 6481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1" is_pre_election: true
I20250111 02:54:01.804572 6406 raft_consensus.cc:2463] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 0.
I20250111 02:54:01.804615 6481 raft_consensus.cc:2463] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 0.
I20250111 02:54:01.804960 6519 leader_election.cc:304] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters:
I20250111 02:54:01.805208 6609 raft_consensus.cc:2798] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:01.805290 6609 raft_consensus.cc:491] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:01.805379 6609 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.807291 6609 raft_consensus.cc:513] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.807878 6609 leader_election.cc:290] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 1 election: Requested vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471)
I20250111 02:54:01.808326 6406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07"
I20250111 02:54:01.808331 6481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1"
I20250111 02:54:01.808499 6406 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.808548 6481 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.810532 6406 raft_consensus.cc:2463] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 1.
I20250111 02:54:01.810534 6481 raft_consensus.cc:2463] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 1.
I20250111 02:54:01.810951 6518 leader_election.cc:304] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:01.811213 6609 raft_consensus.cc:2798] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:01.811359 6609 raft_consensus.cc:695] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 LEADER]: Becoming Leader. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Running, Role: LEADER
I20250111 02:54:01.811625 6609 consensus_queue.cc:237] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.814108 6304 catalog_manager.cc:5526] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16ab852a34d94de69641565f2bbd2881 (127.0.103.3). New cstate: current_term: 1 leader_uuid: "16ab852a34d94de69641565f2bbd2881" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: HEALTHY } } }
I20250111 02:54:01.861938 6603 raft_consensus.cc:491] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:01.862159 6603 raft_consensus.cc:513] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.862972 6603 leader_election.cc:290] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:01.868966 6481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1" is_pre_election: true
I20250111 02:54:01.868966 6555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881" is_pre_election: true
I20250111 02:54:01.869277 6481 raft_consensus.cc:2463] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 0.
I20250111 02:54:01.869279 6555 raft_consensus.cc:2463] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 0.
I20250111 02:54:01.869745 6369 leader_election.cc:304] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:01.870064 6603 raft_consensus.cc:2798] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:01.870152 6603 raft_consensus.cc:491] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:01.870241 6603 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.872032 6603 raft_consensus.cc:513] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.872612 6603 leader_election.cc:290] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 1 election: Requested vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:01.873113 6481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1"
I20250111 02:54:01.873104 6555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881"
I20250111 02:54:01.873322 6481 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.873324 6555 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.876062 6555 raft_consensus.cc:2463] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 1.
I20250111 02:54:01.876062 6481 raft_consensus.cc:2463] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 1.
I20250111 02:54:01.876492 6372 leader_election.cc:304] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters:
I20250111 02:54:01.876792 6603 raft_consensus.cc:2798] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:01.877157 6603 raft_consensus.cc:695] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 LEADER]: Becoming Leader. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Running, Role: LEADER
I20250111 02:54:01.877415 6603 consensus_queue.cc:237] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.879678 6304 catalog_manager.cc:5526] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "bd3f4281bc0f4958b77f4304323bbd07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:01.958343 6603 raft_consensus.cc:491] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:01.958631 6603 raft_consensus.cc:513] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.959327 6603 leader_election.cc:290] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:01.959877 6555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881" is_pre_election: true
I20250111 02:54:01.959880 6481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1" is_pre_election: true
I20250111 02:54:01.960085 6555 raft_consensus.cc:2463] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 0.
I20250111 02:54:01.960124 6481 raft_consensus.cc:2463] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 0.
I20250111 02:54:01.960436 6372 leader_election.cc:304] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters:
I20250111 02:54:01.960675 6603 raft_consensus.cc:2798] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:01.960753 6603 raft_consensus.cc:491] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:01.960817 6603 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.962692 6603 raft_consensus.cc:513] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.963220 6603 leader_election.cc:290] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 1 election: Requested vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:01.963696 6555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881"
I20250111 02:54:01.963696 6481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1"
I20250111 02:54:01.963914 6555 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.963971 6481 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:01.965828 6481 raft_consensus.cc:2463] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 1.
I20250111 02:54:01.965828 6555 raft_consensus.cc:2463] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 1.
I20250111 02:54:01.966305 6372 leader_election.cc:304] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters:
I20250111 02:54:01.966562 6603 raft_consensus.cc:2798] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:01.966670 6603 raft_consensus.cc:695] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 LEADER]: Becoming Leader. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Running, Role: LEADER
I20250111 02:54:01.966910 6603 consensus_queue.cc:237] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:01.969592 6304 catalog_manager.cc:5526] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "bd3f4281bc0f4958b77f4304323bbd07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:02.010834 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:54:02.019265 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:02.019660 412 tablet_replica.cc:331] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: stopping tablet replica
I20250111 02:54:02.019896 412 raft_consensus.cc:2238] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:02.020056 412 raft_consensus.cc:2267] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.020671 412 tablet_replica.cc:331] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: stopping tablet replica
I20250111 02:54:02.020781 412 raft_consensus.cc:2238] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:02.020956 412 pending_rounds.cc:62] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Trying to abort 1 pending ops.
I20250111 02:54:02.020990 412 pending_rounds.cc:69] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112966315899203584 op_type: NO_OP noop_request { }
I20250111 02:54:02.021090 412 raft_consensus.cc:2883] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:54:02.021152 412 raft_consensus.cc:2267] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.021996 412 tablet_replica.cc:331] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: stopping tablet replica
I20250111 02:54:02.022112 412 raft_consensus.cc:2238] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:02.022266 412 pending_rounds.cc:62] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Trying to abort 1 pending ops.
I20250111 02:54:02.022300 412 pending_rounds.cc:69] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112966315532005376 op_type: NO_OP noop_request { }
I20250111 02:54:02.022431 412 raft_consensus.cc:2883] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:54:02.022490 412 raft_consensus.cc:2267] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.023239 412 tablet_replica.cc:331] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: stopping tablet replica
I20250111 02:54:02.023339 412 raft_consensus.cc:2238] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:02.023435 412 raft_consensus.cc:2267] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.027845 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
I20250111 02:54:02.031308 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:54:02.040566 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:02.040908 412 tablet_replica.cc:331] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: stopping tablet replica
I20250111 02:54:02.041102 412 raft_consensus.cc:2238] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:02.041275 412 raft_consensus.cc:2267] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.041967 412 tablet_replica.cc:331] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: stopping tablet replica
I20250111 02:54:02.042104 412 raft_consensus.cc:2238] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:02.042230 412 raft_consensus.cc:2267] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.042845 412 tablet_replica.cc:331] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: stopping tablet replica
I20250111 02:54:02.042970 412 raft_consensus.cc:2238] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:02.043097 412 raft_consensus.cc:2267] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.043568 412 tablet_replica.cc:331] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: stopping tablet replica
I20250111 02:54:02.043685 412 raft_consensus.cc:2238] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:02.043793 412 raft_consensus.cc:2267] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.058100 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:54:02.061465 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:54:02.070065 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:02.070384 412 tablet_replica.cc:331] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: stopping tablet replica
I20250111 02:54:02.070608 412 raft_consensus.cc:2238] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:02.070772 412 raft_consensus.cc:2267] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.071383 412 tablet_replica.cc:331] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: stopping tablet replica
I20250111 02:54:02.071522 412 raft_consensus.cc:2238] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:02.071739 412 pending_rounds.cc:62] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Trying to abort 1 pending ops.
I20250111 02:54:02.071779 412 pending_rounds.cc:69] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112966315262619648 op_type: NO_OP noop_request { }
I20250111 02:54:02.071894 412 raft_consensus.cc:2883] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:54:02.071996 412 raft_consensus.cc:2267] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.072635 412 tablet_replica.cc:331] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: stopping tablet replica
I20250111 02:54:02.072758 412 raft_consensus.cc:2238] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:02.072871 412 raft_consensus.cc:2267] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.073406 412 tablet_replica.cc:331] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: stopping tablet replica
I20250111 02:54:02.073527 412 raft_consensus.cc:2238] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:02.073697 412 pending_rounds.cc:62] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Trying to abort 1 pending ops.
I20250111 02:54:02.073737 412 pending_rounds.cc:69] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112966315013984256 op_type: NO_OP noop_request { }
I20250111 02:54:02.073876 412 raft_consensus.cc:2883] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250111 02:54:02.073974 412 raft_consensus.cc:2267] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:02.078915 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:54:02.082491 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:02.085148 6628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:02.085922 6629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:02.087019 6631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:02.087653 412 server_base.cc:1034] running on GCE node
I20250111 02:54:02.094342 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:02.094480 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:02.094555 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564042094551 us; error 0 us; skew 500 ppm
I20250111 02:54:02.094727 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:02.096073 412 webserver.cc:458] Webserver started at http://127.0.103.4:43257/ using document root <none> and password file <none>
I20250111 02:54:02.096321 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:02.096421 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:02.096565 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:02.097208 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-3-root/instance:
uuid: "47c9f617ebe940cd99b353e491c8d7be"
format_stamp: "Formatted at 2025-01-11 02:54:02 on dist-test-slave-4xrj"
I20250111 02:54:02.099874 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:54:02.101409 6636 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:02.101826 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250111 02:54:02.102020 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-3-root
uuid: "47c9f617ebe940cd99b353e491c8d7be"
format_stamp: "Formatted at 2025-01-11 02:54:02 on dist-test-slave-4xrj"
I20250111 02:54:02.102142 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:02.116254 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:02.116780 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:02.117314 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:02.118191 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:02.118270 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:02.118405 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:02.118463 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:02.129381 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.4:34069
I20250111 02:54:02.129393 6698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.4:34069 every 8 connection(s)
I20250111 02:54:02.136051 6699 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38351
I20250111 02:54:02.136193 6699 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:02.136484 6699 heartbeater.cc:510] Master 127.0.103.62:38351 requested a full tablet report, sending...
I20250111 02:54:02.137256 6304 ts_manager.cc:194] Registered new tserver with Master: 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4:34069)
I20250111 02:54:02.137970 6304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39258
W20250111 02:54:02.261683 6352 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:02.262852 6289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250111 02:54:02.264029 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:02.264137 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:02.265604 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:02.265689 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:02.267024 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:02.267119 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
I20250111 02:54:03.139173 6699 heartbeater.cc:502] Master 127.0.103.62:38351 was elected leader, sending a full tablet report...
W20250111 02:54:03.270834 6352 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:03.272686 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:03.272763 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:03.274006 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:03.274068 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:03.275311 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:03.275377 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:04.278985 6352 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:04.282788 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:04.282876 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:04.284215 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:04.284283 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:04.285591 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:04.285655 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:05.289191 6352 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:05.298991 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:05.299086 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:05.300572 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:05.300669 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:05.302059 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:05.302124 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:06.305804 6352 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:06.309792 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:06.309912 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:06.311393 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:06.311465 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.3:40961: connect: Connection refused (error 111)
W20250111 02:54:06.312768 6352 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:06.312830 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.1:46417: connect: Connection refused (error 111)
W20250111 02:54:07.313654 6352 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver f920acdfe48e48cb9bd1463983b819f1 not available for placement
I20250111 02:54:07.396096 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:07.399426 6718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:07.400434 6719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:07.401265 6721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:07.410099 412 server_base.cc:1034] running on GCE node
I20250111 02:54:07.413197 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:07.413316 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:07.413383 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564047413355 us; error 0 us; skew 500 ppm
I20250111 02:54:07.413580 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:07.419870 412 webserver.cc:458] Webserver started at http://127.0.103.1:39569/ using document root <none> and password file <none>
I20250111 02:54:07.420109 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:07.420169 412 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250111 02:54:07.422118 412 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20250111 02:54:07.423753 6726 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:07.424145 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250111 02:54:07.424255 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "bd3f4281bc0f4958b77f4304323bbd07"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:07.424418 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:07.434731 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:07.435233 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:07.435766 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:07.437036 6733 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
I20250111 02:54:07.444564 412 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250111 02:54:07.444680 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20250111 02:54:07.444795 412 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20250111 02:54:07.446823 6733 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap starting.
I20250111 02:54:07.450896 6733 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:07.451293 6733 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap complete.
I20250111 02:54:07.451539 6733 ts_tablet_manager.cc:1397] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250111 02:54:07.452170 6733 raft_consensus.cc:357] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:07.452397 6733 raft_consensus.cc:738] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.452720 6733 consensus_queue.cc:260] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:07.453405 6733 ts_tablet_manager.cc:1428] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250111 02:54:07.453676 6733 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap starting.
I20250111 02:54:07.454509 412 ts_tablet_manager.cc:610] Registered 4 tablets
I20250111 02:54:07.454603 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.009s sys 0.000s
I20250111 02:54:07.457813 6733 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:07.458261 6733 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap complete.
I20250111 02:54:07.458539 6733 ts_tablet_manager.cc:1397] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250111 02:54:07.459569 6733 raft_consensus.cc:357] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.459803 6733 raft_consensus.cc:738] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.460127 6733 consensus_queue.cc:260] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.460776 6733 ts_tablet_manager.cc:1428] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:07.461007 6733 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap starting.
I20250111 02:54:07.465121 6733 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 02:54:07.465658 6733 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap complete.
I20250111 02:54:07.465960 6733 ts_tablet_manager.cc:1397] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250111 02:54:07.466720 6733 raft_consensus.cc:357] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.467087 6733 raft_consensus.cc:738] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.467406 6733 consensus_queue.cc:260] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.467893 6733 ts_tablet_manager.cc:1428] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:07.468307 6733 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap starting.
I20250111 02:54:07.472132 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:46417
I20250111 02:54:07.472159 6794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:46417 every 8 connection(s)
I20250111 02:54:07.472887 6733 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 02:54:07.473430 6733 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Bootstrap complete.
I20250111 02:54:07.473675 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:54:07.473762 6733 ts_tablet_manager.cc:1397] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20250111 02:54:07.474921 6733 raft_consensus.cc:357] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.475468 6733 raft_consensus.cc:738] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.475956 6733 consensus_queue.cc:260] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.476696 6733 ts_tablet_manager.cc:1428] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20250111 02:54:07.484592 6799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:07.485814 6800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:07.487246 412 server_base.cc:1034] running on GCE node
W20250111 02:54:07.487689 6802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:07.488076 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:07.488175 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:07.488243 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564047488216 us; error 0 us; skew 500 ppm
I20250111 02:54:07.488433 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:07.488924 6795 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38351
I20250111 02:54:07.489070 6795 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:07.489526 6795 heartbeater.cc:510] Master 127.0.103.62:38351 requested a full tablet report, sending...
I20250111 02:54:07.489997 412 webserver.cc:458] Webserver started at http://127.0.103.2:46045/ using document root <none> and password file <none>
I20250111 02:54:07.490252 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:07.490334 412 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250111 02:54:07.491135 6304 ts_manager.cc:194] Re-registered known tserver with Master: bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417)
I20250111 02:54:07.492383 412 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20250111 02:54:07.493108 6304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48998
I20250111 02:54:07.494007 6807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:07.494446 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:54:07.494506 6795 heartbeater.cc:502] Master 127.0.103.62:38351 was elected leader, sending a full tablet report...
I20250111 02:54:07.494555 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "f920acdfe48e48cb9bd1463983b819f1"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:07.494719 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:07.528693 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:07.529245 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:07.529819 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:07.538380 412 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250111 02:54:07.538486 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.001s
I20250111 02:54:07.540283 6814 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Bootstrap starting.
I20250111 02:54:07.543802 6814 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:07.544155 6814 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Bootstrap complete.
I20250111 02:54:07.544394 6814 ts_tablet_manager.cc:1397] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:54:07.544799 412 ts_tablet_manager.cc:610] Registered 4 tablets
I20250111 02:54:07.544873 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.002s sys 0.002s
I20250111 02:54:07.545043 6814 raft_consensus.cc:357] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:07.545300 6814 raft_consensus.cc:738] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.545572 6814 consensus_queue.cc:260] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:07.546298 6814 ts_tablet_manager.cc:1428] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250111 02:54:07.546545 6814 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Bootstrap starting.
I20250111 02:54:07.550468 6814 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:07.550983 6814 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Bootstrap complete.
I20250111 02:54:07.551314 6814 ts_tablet_manager.cc:1397] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250111 02:54:07.552239 6814 raft_consensus.cc:357] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.552531 6814 raft_consensus.cc:738] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.552891 6814 consensus_queue.cc:260] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.553517 6814 ts_tablet_manager.cc:1428] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250111 02:54:07.553825 6814 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Bootstrap starting.
I20250111 02:54:07.558691 6814 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:07.559072 6814 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Bootstrap complete.
I20250111 02:54:07.559298 6814 ts_tablet_manager.cc:1397] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250111 02:54:07.559890 6814 raft_consensus.cc:357] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.560071 6814 raft_consensus.cc:738] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.560307 6814 consensus_queue.cc:260] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.560742 6814 ts_tablet_manager.cc:1428] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:54:07.560967 6814 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Bootstrap starting.
I20250111 02:54:07.562858 6874 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:43471 every 8 connection(s)
I20250111 02:54:07.562858 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:43471
I20250111 02:54:07.564399 6814 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:07.564473 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:54:07.564851 6814 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Bootstrap complete.
I20250111 02:54:07.565163 6814 ts_tablet_manager.cc:1397] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250111 02:54:07.566069 6814 raft_consensus.cc:357] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.566493 6814 raft_consensus.cc:738] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.566835 6814 consensus_queue.cc:260] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.567438 6814 ts_tablet_manager.cc:1428] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250111 02:54:07.568694 6880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:07.569826 412 server_base.cc:1034] running on GCE node
W20250111 02:54:07.570075 6879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:07.571197 6882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:07.571668 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:07.571767 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:07.571831 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564047571809 us; error 0 us; skew 500 ppm
I20250111 02:54:07.572054 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:07.572160 6876 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38351
I20250111 02:54:07.572294 6876 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:07.572676 6876 heartbeater.cc:510] Master 127.0.103.62:38351 requested a full tablet report, sending...
I20250111 02:54:07.573410 412 webserver.cc:458] Webserver started at http://127.0.103.3:33311/ using document root <none> and password file <none>
I20250111 02:54:07.573689 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:07.573776 412 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250111 02:54:07.574191 6304 ts_manager.cc:194] Re-registered known tserver with Master: f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471)
I20250111 02:54:07.575776 412 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:54:07.576107 6304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49004
I20250111 02:54:07.577421 6876 heartbeater.cc:502] Master 127.0.103.62:38351 was elected leader, sending a full tablet report...
I20250111 02:54:07.577490 6887 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:07.577889 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250111 02:54:07.578011 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "16ab852a34d94de69641565f2bbd2881"
format_stamp: "Formatted at 2025-01-11 02:54:00 on dist-test-slave-4xrj"
I20250111 02:54:07.578145 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:07.590287 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:07.590821 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:07.591336 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:07.599798 412 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250111 02:54:07.599934 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20250111 02:54:07.601894 6894 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Bootstrap starting.
I20250111 02:54:07.606156 6894 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 02:54:07.606544 6894 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Bootstrap complete.
I20250111 02:54:07.606781 6894 ts_tablet_manager.cc:1397] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20250111 02:54:07.606891 412 ts_tablet_manager.cc:610] Registered 4 tablets
I20250111 02:54:07.606988 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.006s sys 0.000s
I20250111 02:54:07.607542 6894 raft_consensus.cc:357] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:07.607795 6894 raft_consensus.cc:738] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.608042 6894 consensus_queue.cc:260] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:07.608847 6894 ts_tablet_manager.cc:1428] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:07.609131 6894 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Bootstrap starting.
I20250111 02:54:07.614027 6894 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250111 02:54:07.614573 6894 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Bootstrap complete.
I20250111 02:54:07.614846 6894 ts_tablet_manager.cc:1397] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20250111 02:54:07.615931 6894 raft_consensus.cc:357] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.616369 6894 raft_consensus.cc:738] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.616647 6894 consensus_queue.cc:260] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.617282 6894 ts_tablet_manager.cc:1428] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250111 02:54:07.617507 6894 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Bootstrap starting.
I20250111 02:54:07.622027 6894 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:07.622593 6894 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Bootstrap complete.
I20250111 02:54:07.622979 6894 ts_tablet_manager.cc:1397] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250111 02:54:07.623936 6894 raft_consensus.cc:357] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.624238 6894 raft_consensus.cc:738] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.624580 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:40961
I20250111 02:54:07.624590 6954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:40961 every 8 connection(s)
I20250111 02:54:07.624572 6894 consensus_queue.cc:260] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.625166 6894 ts_tablet_manager.cc:1428] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250111 02:54:07.625509 6894 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Bootstrap starting.
I20250111 02:54:07.629108 6894 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:07.629501 6894 tablet_bootstrap.cc:492] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Bootstrap complete.
I20250111 02:54:07.629807 6894 ts_tablet_manager.cc:1397] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:54:07.630452 6894 raft_consensus.cc:357] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.630689 6894 raft_consensus.cc:738] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Initialized, Role: FOLLOWER
I20250111 02:54:07.631255 6894 consensus_queue.cc:260] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:07.631701 6894 ts_tablet_manager.cc:1428] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250111 02:54:07.632385 6955 heartbeater.cc:346] Connected to a master server at 127.0.103.62:38351
I20250111 02:54:07.632550 6955 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:07.632870 6955 heartbeater.cc:510] Master 127.0.103.62:38351 requested a full tablet report, sending...
I20250111 02:54:07.634013 6304 ts_manager.cc:194] Re-registered known tserver with Master: 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:07.635675 6304 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49014
I20250111 02:54:07.636754 6955 heartbeater.cc:502] Master 127.0.103.62:38351 was elected leader, sending a full tablet report...
W20250111 02:54:08.322911 6352 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 16ab852a34d94de69641565f2bbd2881 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } }
W20250111 02:54:08.324352 6352 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d9c4f53f3a2e4b49b9ef215ea5454686, TS bd3f4281bc0f4958b77f4304323bbd07 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:08.324465 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d9c4f53f3a2e4b49b9ef215ea5454686, TS bd3f4281bc0f4958b77f4304323bbd07 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:08.330659 6352 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3f4cf57ca748467ead76333d1e58a20f, TS f920acdfe48e48cb9bd1463983b819f1 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:08.330777 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 3f4cf57ca748467ead76333d1e58a20f, TS f920acdfe48e48cb9bd1463983b819f1 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:08.332020 6352 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0741856ace474541b370b5579ab9093b, TS 16ab852a34d94de69641565f2bbd2881 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:08.332110 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0741856ace474541b370b5579ab9093b, TS 16ab852a34d94de69641565f2bbd2881 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.706430 6962 raft_consensus.cc:491] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:08.706636 6962 raft_consensus.cc:513] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.707522 6962 leader_election.cc:290] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:08.707789 6962 raft_consensus.cc:491] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:08.708057 6962 raft_consensus.cc:513] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.709595 6962 leader_election.cc:290] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:08.715750 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881" is_pre_election: true
I20250111 02:54:08.715843 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07" is_pre_election: true
I20250111 02:54:08.716046 6929 raft_consensus.cc:2463] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f920acdfe48e48cb9bd1463983b819f1 in term 1.
I20250111 02:54:08.715874 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07" is_pre_election: true
I20250111 02:54:08.716240 6769 raft_consensus.cc:2405] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f920acdfe48e48cb9bd1463983b819f1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250111 02:54:08.716318 6768 raft_consensus.cc:2405] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f920acdfe48e48cb9bd1463983b819f1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250111 02:54:08.715837 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881" is_pre_election: true
I20250111 02:54:08.716567 6811 leader_election.cc:304] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:08.716678 6928 raft_consensus.cc:2463] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f920acdfe48e48cb9bd1463983b819f1 in term 1.
I20250111 02:54:08.716858 6962 raft_consensus.cc:2798] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:54:08.716996 6811 leader_election.cc:304] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, f920acdfe48e48cb9bd1463983b819f1; no voters: bd3f4281bc0f4958b77f4304323bbd07
I20250111 02:54:08.716995 6962 raft_consensus.cc:491] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:08.717150 6962 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.717265 6963 raft_consensus.cc:2798] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:54:08.717375 6963 raft_consensus.cc:491] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:08.717485 6963 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.719278 6962 raft_consensus.cc:513] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.719388 6963 raft_consensus.cc:513] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.719975 6963 leader_election.cc:290] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 election: Requested vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:08.719866 6962 leader_election.cc:290] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 election: Requested vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:08.720281 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07"
I20250111 02:54:08.720393 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881"
I20250111 02:54:08.720502 6768 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.720541 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881"
I20250111 02:54:08.720569 6928 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.720566 6769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07"
I20250111 02:54:08.720788 6929 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.720898 6769 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.723446 6928 raft_consensus.cc:2463] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f920acdfe48e48cb9bd1463983b819f1 in term 2.
I20250111 02:54:08.723546 6768 raft_consensus.cc:2405] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f920acdfe48e48cb9bd1463983b819f1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250111 02:54:08.723639 6769 raft_consensus.cc:2405] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f920acdfe48e48cb9bd1463983b819f1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250111 02:54:08.723556 6929 raft_consensus.cc:2463] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f920acdfe48e48cb9bd1463983b819f1 in term 2.
I20250111 02:54:08.723810 6811 leader_election.cc:304] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:08.724095 6962 raft_consensus.cc:2798] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:54:08.724254 6962 raft_consensus.cc:695] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Becoming Leader. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Running, Role: LEADER
I20250111 02:54:08.724260 6811 leader_election.cc:304] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, f920acdfe48e48cb9bd1463983b819f1; no voters: bd3f4281bc0f4958b77f4304323bbd07
I20250111 02:54:08.724568 6963 raft_consensus.cc:2798] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:54:08.724596 6962 consensus_queue.cc:237] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.725065 6963 raft_consensus.cc:695] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Becoming Leader. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Running, Role: LEADER
I20250111 02:54:08.725324 6963 consensus_queue.cc:237] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.727450 6304 catalog_manager.cc:5526] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 reported cstate change: term changed from 1 to 2, leader changed from bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1) to f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2). New cstate: current_term: 2 leader_uuid: "f920acdfe48e48cb9bd1463983b819f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:08.732503 6304 catalog_manager.cc:5526] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 reported cstate change: term changed from 1 to 2, leader changed from bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1) to f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2). New cstate: current_term: 2 leader_uuid: "f920acdfe48e48cb9bd1463983b819f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:08.802649 6973 raft_consensus.cc:491] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:08.802886 6973 raft_consensus.cc:513] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.803687 6973 leader_election.cc:290] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:08.810230 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881" is_pre_election: true
I20250111 02:54:08.810238 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1" is_pre_election: true
I20250111 02:54:08.810488 6929 raft_consensus.cc:2463] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 2.
I20250111 02:54:08.810868 6730 leader_election.cc:304] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters: f920acdfe48e48cb9bd1463983b819f1
I20250111 02:54:08.811120 6973 raft_consensus.cc:2798] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250111 02:54:08.811218 6973 raft_consensus.cc:491] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:08.811303 6973 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:08.813092 6973 raft_consensus.cc:513] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.813604 6973 leader_election.cc:290] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 3 election: Requested vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:08.814031 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1"
I20250111 02:54:08.814072 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0741856ace474541b370b5579ab9093b" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881"
I20250111 02:54:08.814216 6929 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:08.816263 6929 raft_consensus.cc:2463] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 3.
I20250111 02:54:08.816653 6730 leader_election.cc:304] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters: f920acdfe48e48cb9bd1463983b819f1
I20250111 02:54:08.816919 6973 raft_consensus.cc:2798] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:54:08.817297 6973 raft_consensus.cc:695] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 3 LEADER]: Becoming Leader. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Running, Role: LEADER
I20250111 02:54:08.817560 6973 consensus_queue.cc:237] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.819808 6304 catalog_manager.cc:5526] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 reported cstate change: term changed from 2 to 3, leader changed from f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2) to bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1). New cstate: current_term: 3 leader_uuid: "bd3f4281bc0f4958b77f4304323bbd07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:08.852227 6979 raft_consensus.cc:491] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:08.852416 6979 raft_consensus.cc:513] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:08.853191 6979 leader_election.cc:290] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417)
I20250111 02:54:08.859496 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486aa5274c4b4f3aa94a5c82682d146f" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1" is_pre_election: true
I20250111 02:54:08.859575 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486aa5274c4b4f3aa94a5c82682d146f" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07" is_pre_election: true
I20250111 02:54:08.859737 6849 raft_consensus.cc:2463] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 1.
I20250111 02:54:08.859778 6768 raft_consensus.cc:2463] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 1.
I20250111 02:54:08.860169 6888 leader_election.cc:304] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:08.860419 6979 raft_consensus.cc:2798] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:54:08.860508 6979 raft_consensus.cc:491] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:08.860598 6979 raft_consensus.cc:3054] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.862607 6979 raft_consensus.cc:513] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:08.863134 6979 leader_election.cc:290] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 2 election: Requested vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417)
I20250111 02:54:08.863564 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486aa5274c4b4f3aa94a5c82682d146f" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07"
I20250111 02:54:08.863564 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "486aa5274c4b4f3aa94a5c82682d146f" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1"
I20250111 02:54:08.863795 6768 raft_consensus.cc:3054] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.863824 6849 raft_consensus.cc:3054] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.866412 6768 raft_consensus.cc:2463] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 2.
I20250111 02:54:08.866412 6849 raft_consensus.cc:2463] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 2.
I20250111 02:54:08.866855 6889 leader_election.cc:304] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:08.867146 6979 raft_consensus.cc:2798] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:54:08.867506 6979 raft_consensus.cc:695] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 2 LEADER]: Becoming Leader. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Running, Role: LEADER
I20250111 02:54:08.867762 6979 consensus_queue.cc:237] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:08.870050 6304 catalog_manager.cc:5526] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "16ab852a34d94de69641565f2bbd2881" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:08.879659 6963 raft_consensus.cc:491] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:08.879833 6963 raft_consensus.cc:513] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.880399 6963 leader_election.cc:290] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:08.880837 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07" is_pre_election: true
I20250111 02:54:08.880837 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881" is_pre_election: true
I20250111 02:54:08.881089 6768 raft_consensus.cc:2463] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f920acdfe48e48cb9bd1463983b819f1 in term 1.
I20250111 02:54:08.881153 6929 raft_consensus.cc:2405] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f920acdfe48e48cb9bd1463983b819f1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250111 02:54:08.881480 6809 leader_election.cc:304] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3f4281bc0f4958b77f4304323bbd07, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:08.881757 6963 raft_consensus.cc:2798] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:54:08.881871 6963 raft_consensus.cc:491] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:08.881968 6963 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.883792 6963 raft_consensus.cc:513] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.884277 6963 leader_election.cc:290] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 election: Requested vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:08.884680 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881"
I20250111 02:54:08.884708 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "f920acdfe48e48cb9bd1463983b819f1" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07"
I20250111 02:54:08.884853 6929 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.884858 6768 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:08.886713 6768 raft_consensus.cc:2463] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f920acdfe48e48cb9bd1463983b819f1 in term 2.
I20250111 02:54:08.886811 6929 raft_consensus.cc:2405] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f920acdfe48e48cb9bd1463983b819f1 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250111 02:54:08.887079 6809 leader_election.cc:304] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3f4281bc0f4958b77f4304323bbd07, f920acdfe48e48cb9bd1463983b819f1; no voters:
I20250111 02:54:08.887315 6963 raft_consensus.cc:2798] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:54:08.887461 6963 raft_consensus.cc:695] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Becoming Leader. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Running, Role: LEADER
I20250111 02:54:08.887689 6963 consensus_queue.cc:237] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:08.889989 6304 catalog_manager.cc:5526] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 reported cstate change: term changed from 1 to 2, leader changed from 16ab852a34d94de69641565f2bbd2881 (127.0.103.3) to f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2). New cstate: current_term: 2 leader_uuid: "f920acdfe48e48cb9bd1463983b819f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:09.238965 6929 raft_consensus.cc:1235] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Rejecting Update request from peer f920acdfe48e48cb9bd1463983b819f1 for earlier term 2. Current term is 3. Ops: []
I20250111 02:54:09.239549 6963 consensus_queue.cc:1046] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.239817 6962 raft_consensus.cc:3049] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:54:09.239919 6962 raft_consensus.cc:738] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Running, Role: LEADER
I20250111 02:54:09.240177 6962 consensus_queue.cc:260] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.240446 6962 raft_consensus.cc:3054] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:09.244935 6979 raft_consensus.cc:491] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:09.245081 6979 raft_consensus.cc:513] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.245690 6979 leader_election.cc:290] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471)
I20250111 02:54:09.246188 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1" is_pre_election: true
I20250111 02:54:09.246186 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07" is_pre_election: true
I20250111 02:54:09.246431 6768 raft_consensus.cc:2463] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 2.
I20250111 02:54:09.246805 6889 leader_election.cc:304] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters: f920acdfe48e48cb9bd1463983b819f1
I20250111 02:54:09.247032 6979 raft_consensus.cc:2798] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250111 02:54:09.247124 6979 raft_consensus.cc:491] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:09.247202 6979 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:09.248950 6979 raft_consensus.cc:513] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.249501 6979 leader_election.cc:290] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 3 election: Requested vote from peers bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417), f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471)
I20250111 02:54:09.249935 6768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bd3f4281bc0f4958b77f4304323bbd07"
I20250111 02:54:09.249931 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9c4f53f3a2e4b49b9ef215ea5454686" candidate_uuid: "16ab852a34d94de69641565f2bbd2881" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1"
I20250111 02:54:09.250167 6768 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:09.251812 6768 raft_consensus.cc:2463] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ab852a34d94de69641565f2bbd2881 in term 3.
I20250111 02:54:09.252106 6889 leader_election.cc:304] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters: f920acdfe48e48cb9bd1463983b819f1
I20250111 02:54:09.252337 6979 raft_consensus.cc:2798] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:54:09.252482 6979 raft_consensus.cc:695] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 3 LEADER]: Becoming Leader. State: Replica: 16ab852a34d94de69641565f2bbd2881, State: Running, Role: LEADER
I20250111 02:54:09.252709 6979 consensus_queue.cc:237] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.254947 6304 catalog_manager.cc:5526] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 reported cstate change: term changed from 2 to 3, leader changed from f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2) to 16ab852a34d94de69641565f2bbd2881 (127.0.103.3). New cstate: current_term: 3 leader_uuid: "16ab852a34d94de69641565f2bbd2881" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: HEALTHY } } }
I20250111 02:54:09.263976 6973 raft_consensus.cc:491] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:09.264154 6973 raft_consensus.cc:513] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.264732 6973 leader_election.cc:290] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:09.265177 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1" is_pre_election: true
I20250111 02:54:09.265183 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881" is_pre_election: true
I20250111 02:54:09.265460 6929 raft_consensus.cc:2463] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 2.
I20250111 02:54:09.265892 6730 leader_election.cc:304] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters: f920acdfe48e48cb9bd1463983b819f1
I20250111 02:54:09.266135 6973 raft_consensus.cc:2798] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250111 02:54:09.266214 6973 raft_consensus.cc:491] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:09.266307 6973 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:09.268136 6973 raft_consensus.cc:513] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.268926 6973 leader_election.cc:290] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 3 election: Requested vote from peers f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2:43471), 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:09.269028 6929 raft_consensus.cc:1270] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Refusing update from remote peer f920acdfe48e48cb9bd1463983b819f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250111 02:54:09.269239 6849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f920acdfe48e48cb9bd1463983b819f1"
I20250111 02:54:09.269237 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f4cf57ca748467ead76333d1e58a20f" candidate_uuid: "bd3f4281bc0f4958b77f4304323bbd07" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "16ab852a34d94de69641565f2bbd2881"
I20250111 02:54:09.269516 6928 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:09.269552 6962 consensus_queue.cc:1035] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.271595 6928 raft_consensus.cc:2463] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3f4281bc0f4958b77f4304323bbd07 in term 3.
I20250111 02:54:09.271831 6929 raft_consensus.cc:1235] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Rejecting Update request from peer f920acdfe48e48cb9bd1463983b819f1 for earlier term 2. Current term is 3. Ops: [2.1-2.1]
I20250111 02:54:09.271939 6730 leader_election.cc:304] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16ab852a34d94de69641565f2bbd2881, bd3f4281bc0f4958b77f4304323bbd07; no voters: f920acdfe48e48cb9bd1463983b819f1
I20250111 02:54:09.272212 6973 raft_consensus.cc:2798] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:54:09.272497 6973 raft_consensus.cc:695] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 3 LEADER]: Becoming Leader. State: Replica: bd3f4281bc0f4958b77f4304323bbd07, State: Running, Role: LEADER
I20250111 02:54:09.272629 6962 consensus_queue.cc:1046] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.272876 6963 raft_consensus.cc:3049] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:54:09.272776 6973 consensus_queue.cc:237] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.272964 6963 raft_consensus.cc:738] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Running, Role: LEADER
I20250111 02:54:09.273182 6963 consensus_queue.cc:260] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.273480 6963 raft_consensus.cc:3054] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:09.275365 6304 catalog_manager.cc:5526] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 reported cstate change: term changed from 2 to 3, leader changed from f920acdfe48e48cb9bd1463983b819f1 (127.0.103.2) to bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1). New cstate: current_term: 3 leader_uuid: "bd3f4281bc0f4958b77f4304323bbd07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: UNKNOWN } } }
W20250111 02:54:09.335718 6352 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20250111 02:54:09.337080 6352 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3f4cf57ca748467ead76333d1e58a20f, TS 16ab852a34d94de69641565f2bbd2881 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:09.337154 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 3f4cf57ca748467ead76333d1e58a20f, TS 16ab852a34d94de69641565f2bbd2881 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:09.338312 6352 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0741856ace474541b370b5579ab9093b, TS f920acdfe48e48cb9bd1463983b819f1 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:09.338389 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0741856ace474541b370b5579ab9093b, TS f920acdfe48e48cb9bd1463983b819f1 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
W20250111 02:54:09.339711 6352 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 486aa5274c4b4f3aa94a5c82682d146f, TS bd3f4281bc0f4958b77f4304323bbd07 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
W20250111 02:54:09.339776 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 486aa5274c4b4f3aa94a5c82682d146f, TS bd3f4281bc0f4958b77f4304323bbd07 -> TS 47c9f617ebe940cd99b353e491c8d7be move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } }
I20250111 02:54:09.351718 6849 raft_consensus.cc:1270] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Refusing update from remote peer bd3f4281bc0f4958b77f4304323bbd07: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250111 02:54:09.352299 6973 consensus_queue.cc:1035] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.352553 6973 consensus_queue.cc:1225] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Peer f920acdfe48e48cb9bd1463983b819f1 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250111 02:54:09.353678 6849 pending_rounds.cc:77] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Aborting all ops after (but not including) 0
I20250111 02:54:09.353755 6849 pending_rounds.cc:99] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250111 02:54:09.353819 6849 raft_consensus.cc:2883] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 02:54:09.356827 6929 raft_consensus.cc:1270] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Refusing update from remote peer bd3f4281bc0f4958b77f4304323bbd07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250111 02:54:09.357278 6977 consensus_queue.cc:1035] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.394834 6929 raft_consensus.cc:1235] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 3 LEADER]: Rejecting Update request from peer f920acdfe48e48cb9bd1463983b819f1 for earlier term 2. Current term is 3. Ops: []
I20250111 02:54:09.395375 6963 consensus_queue.cc:1046] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.395624 6963 consensus_queue.cc:1225] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [LEADER]: Peer 16ab852a34d94de69641565f2bbd2881 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250111 02:54:09.395732 6962 raft_consensus.cc:3049] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:54:09.395814 6962 raft_consensus.cc:738] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f920acdfe48e48cb9bd1463983b819f1, State: Running, Role: LEADER
I20250111 02:54:09.396080 6962 consensus_queue.cc:260] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } }
I20250111 02:54:09.396354 6962 raft_consensus.cc:3054] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:09.486094 6849 raft_consensus.cc:1270] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Refusing update from remote peer 16ab852a34d94de69641565f2bbd2881: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:54:09.486701 6979 consensus_queue.cc:1035] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.490342 6768 raft_consensus.cc:1270] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Refusing update from remote peer 16ab852a34d94de69641565f2bbd2881: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:54:09.491405 6993 consensus_queue.cc:1035] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.732056 6768 raft_consensus.cc:1270] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Refusing update from remote peer 16ab852a34d94de69641565f2bbd2881: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250111 02:54:09.732582 6993 consensus_queue.cc:1035] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.736179 6849 raft_consensus.cc:1270] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Refusing update from remote peer 16ab852a34d94de69641565f2bbd2881: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250111 02:54:09.736907 6982 consensus_queue.cc:1035] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.737167 6982 consensus_queue.cc:1225] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Peer f920acdfe48e48cb9bd1463983b819f1 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250111 02:54:09.738185 6849 pending_rounds.cc:77] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Aborting all ops after (but not including) 0
I20250111 02:54:09.738245 6849 pending_rounds.cc:99] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250111 02:54:09.738291 6849 raft_consensus.cc:2883] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 02:54:09.763185 6849 raft_consensus.cc:1270] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Refusing update from remote peer bd3f4281bc0f4958b77f4304323bbd07: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250111 02:54:09.763761 6977 consensus_queue.cc:1035] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:09.763993 6977 consensus_queue.cc:1225] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Peer f920acdfe48e48cb9bd1463983b819f1 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250111 02:54:09.764961 6849 pending_rounds.cc:77] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Aborting all ops after (but not including) 0
I20250111 02:54:09.765025 6849 pending_rounds.cc:99] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250111 02:54:09.765075 6849 raft_consensus.cc:2883] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250111 02:54:09.768041 6929 raft_consensus.cc:1270] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Refusing update from remote peer bd3f4281bc0f4958b77f4304323bbd07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250111 02:54:09.768694 6988 consensus_queue.cc:1035] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:10.343537 6929 consensus_queue.cc:237] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
I20250111 02:54:10.346314 6768 raft_consensus.cc:1270] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Refusing update from remote peer 16ab852a34d94de69641565f2bbd2881: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:54:10.346897 6849 raft_consensus.cc:1270] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Refusing update from remote peer 16ab852a34d94de69641565f2bbd2881: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:54:10.346810 6979 consensus_queue.cc:1035] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:10.347483 6979 consensus_queue.cc:1035] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:10.349138 6993 raft_consensus.cc:2949] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
I20250111 02:54:10.350883 6849 raft_consensus.cc:2949] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
I20250111 02:54:10.352507 6768 raft_consensus.cc:2949] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
W20250111 02:54:10.353029 6891 consensus_peers.cc:487] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 -> Peer 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4:34069): Couldn't send request to peer 47c9f617ebe940cd99b353e491c8d7be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 486aa5274c4b4f3aa94a5c82682d146f. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:10.354863 6304 catalog_manager.cc:5526] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 reported cstate change: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New cstate: current_term: 2 leader_uuid: "16ab852a34d94de69641565f2bbd2881" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:10.355525 6929 consensus_queue.cc:237] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } attrs { replace: true } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
W20250111 02:54:10.358728 6891 consensus_peers.cc:487] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 -> Peer 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4:34069): Couldn't send request to peer 47c9f617ebe940cd99b353e491c8d7be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9c4f53f3a2e4b49b9ef215ea5454686. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:10.358759 6849 raft_consensus.cc:1270] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Refusing update from remote peer 16ab852a34d94de69641565f2bbd2881: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250111 02:54:10.359061 6768 raft_consensus.cc:1270] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Refusing update from remote peer 16ab852a34d94de69641565f2bbd2881: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250111 02:54:10.359185 6979 consensus_queue.cc:1035] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:10.359541 6982 consensus_queue.cc:1035] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:10.361917 6982 raft_consensus.cc:2949] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } attrs { replace: true } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
I20250111 02:54:10.363137 6849 raft_consensus.cc:2949] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } attrs { replace: true } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
I20250111 02:54:10.365633 6768 raft_consensus.cc:2949] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } attrs { replace: true } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
I20250111 02:54:10.367652 6303 catalog_manager.cc:5526] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 reported cstate change: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New cstate: current_term: 3 leader_uuid: "16ab852a34d94de69641565f2bbd2881" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } attrs { replace: true } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
I20250111 02:54:10.370453 6768 consensus_queue.cc:237] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } attrs { replace: true } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
I20250111 02:54:10.372695 6849 raft_consensus.cc:1270] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Refusing update from remote peer bd3f4281bc0f4958b77f4304323bbd07: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250111 02:54:10.373250 6973 consensus_queue.cc:1035] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:10.376026 6988 raft_consensus.cc:2949] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } attrs { replace: true } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
I20250111 02:54:10.376663 6849 raft_consensus.cc:2949] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } attrs { replace: true } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
I20250111 02:54:10.380470 6304 catalog_manager.cc:5526] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 reported cstate change: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New cstate: current_term: 3 leader_uuid: "bd3f4281bc0f4958b77f4304323bbd07" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:10.385528 6929 raft_consensus.cc:1270] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Refusing update from remote peer bd3f4281bc0f4958b77f4304323bbd07: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20250111 02:54:10.386041 6977 consensus_queue.cc:1035] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:10.388816 6929 raft_consensus.cc:2949] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } attrs { replace: true } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } } }
W20250111 02:54:10.391206 6730 consensus_peers.cc:487] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 -> Peer 47c9f617ebe940cd99b353e491c8d7be (127.0.103.4:34069): Couldn't send request to peer 47c9f617ebe940cd99b353e491c8d7be. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0741856ace474541b370b5579ab9093b. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:10.876753 7009 ts_tablet_manager.cc:927] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: Initiating tablet copy from peer 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:10.877293 7009 tablet_copy_client.cc:323] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:40961
I20250111 02:54:10.883518 6939 tablet_copy_service.cc:140] P 16ab852a34d94de69641565f2bbd2881: Received BeginTabletCopySession request for tablet 486aa5274c4b4f3aa94a5c82682d146f from peer 47c9f617ebe940cd99b353e491c8d7be ({username='slave'} at 127.0.0.1:57338)
I20250111 02:54:10.883708 6939 tablet_copy_service.cc:161] P 16ab852a34d94de69641565f2bbd2881: Beginning new tablet copy session on tablet 486aa5274c4b4f3aa94a5c82682d146f from peer 47c9f617ebe940cd99b353e491c8d7be at {username='slave'} at 127.0.0.1:57338: session id = 47c9f617ebe940cd99b353e491c8d7be-486aa5274c4b4f3aa94a5c82682d146f
I20250111 02:54:10.886276 6939 tablet_copy_source_session.cc:215] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:54:10.887414 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 486aa5274c4b4f3aa94a5c82682d146f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:10.892475 7009 tablet_copy_client.cc:806] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Starting download of 0 data blocks...
I20250111 02:54:10.892724 7009 tablet_copy_client.cc:670] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Starting download of 1 WAL segments...
I20250111 02:54:10.894061 7009 tablet_copy_client.cc:538] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:54:10.896885 7009 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap starting.
I20250111 02:54:10.902024 7009 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:10.902369 7009 tablet_bootstrap.cc:492] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap complete.
I20250111 02:54:10.902577 7009 ts_tablet_manager.cc:1397] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: Time spent bootstrapping tablet: real 0.006s user 0.008s sys 0.000s
I20250111 02:54:10.903206 7009 raft_consensus.cc:357] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
I20250111 02:54:10.903472 7009 raft_consensus.cc:738] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 47c9f617ebe940cd99b353e491c8d7be, State: Initialized, Role: LEARNER
I20250111 02:54:10.903646 7009 consensus_queue.cc:260] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } attrs { replace: true } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
I20250111 02:54:10.904275 7009 ts_tablet_manager.cc:1428] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:10.904938 6939 tablet_copy_service.cc:342] P 16ab852a34d94de69641565f2bbd2881: Request end of tablet copy session 47c9f617ebe940cd99b353e491c8d7be-486aa5274c4b4f3aa94a5c82682d146f received from {username='slave'} at 127.0.0.1:57338
I20250111 02:54:10.905045 6939 tablet_copy_service.cc:434] P 16ab852a34d94de69641565f2bbd2881: ending tablet copy session 47c9f617ebe940cd99b353e491c8d7be-486aa5274c4b4f3aa94a5c82682d146f on tablet 486aa5274c4b4f3aa94a5c82682d146f with peer 47c9f617ebe940cd99b353e491c8d7be
I20250111 02:54:10.908116 7009 ts_tablet_manager.cc:927] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: Initiating tablet copy from peer 16ab852a34d94de69641565f2bbd2881 (127.0.103.3:40961)
I20250111 02:54:10.908603 7009 tablet_copy_client.cc:323] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:40961
I20250111 02:54:10.909097 6939 tablet_copy_service.cc:140] P 16ab852a34d94de69641565f2bbd2881: Received BeginTabletCopySession request for tablet d9c4f53f3a2e4b49b9ef215ea5454686 from peer 47c9f617ebe940cd99b353e491c8d7be ({username='slave'} at 127.0.0.1:57338)
I20250111 02:54:10.909243 6939 tablet_copy_service.cc:161] P 16ab852a34d94de69641565f2bbd2881: Beginning new tablet copy session on tablet d9c4f53f3a2e4b49b9ef215ea5454686 from peer 47c9f617ebe940cd99b353e491c8d7be at {username='slave'} at 127.0.0.1:57338: session id = 47c9f617ebe940cd99b353e491c8d7be-d9c4f53f3a2e4b49b9ef215ea5454686
I20250111 02:54:10.911551 6939 tablet_copy_source_session.cc:215] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:54:10.912477 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9c4f53f3a2e4b49b9ef215ea5454686. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:10.917157 7009 tablet_copy_client.cc:806] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Starting download of 0 data blocks...
I20250111 02:54:10.917398 7009 tablet_copy_client.cc:670] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Starting download of 1 WAL segments...
I20250111 02:54:10.918756 7009 tablet_copy_client.cc:538] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:54:10.921638 7009 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap starting.
I20250111 02:54:10.927573 7009 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:10.927922 7009 tablet_bootstrap.cc:492] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap complete.
I20250111 02:54:10.928146 7009 ts_tablet_manager.cc:1397] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20250111 02:54:10.928771 7009 raft_consensus.cc:357] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } attrs { replace: true } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
I20250111 02:54:10.928999 7009 raft_consensus.cc:738] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 47c9f617ebe940cd99b353e491c8d7be, State: Initialized, Role: LEARNER
I20250111 02:54:10.929209 7009 consensus_queue.cc:260] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } attrs { replace: true } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
I20250111 02:54:10.930240 7009 ts_tablet_manager.cc:1428] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250111 02:54:10.931033 6939 tablet_copy_service.cc:342] P 16ab852a34d94de69641565f2bbd2881: Request end of tablet copy session 47c9f617ebe940cd99b353e491c8d7be-d9c4f53f3a2e4b49b9ef215ea5454686 received from {username='slave'} at 127.0.0.1:57338
I20250111 02:54:10.931121 6939 tablet_copy_service.cc:434] P 16ab852a34d94de69641565f2bbd2881: ending tablet copy session 47c9f617ebe940cd99b353e491c8d7be-d9c4f53f3a2e4b49b9ef215ea5454686 on tablet d9c4f53f3a2e4b49b9ef215ea5454686 with peer 47c9f617ebe940cd99b353e491c8d7be
I20250111 02:54:10.942256 7009 ts_tablet_manager.cc:927] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: Initiating tablet copy from peer bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417)
I20250111 02:54:10.942823 7009 tablet_copy_client.cc:323] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.1:46417
I20250111 02:54:10.949028 6779 tablet_copy_service.cc:140] P bd3f4281bc0f4958b77f4304323bbd07: Received BeginTabletCopySession request for tablet 0741856ace474541b370b5579ab9093b from peer 47c9f617ebe940cd99b353e491c8d7be ({username='slave'} at 127.0.0.1:54304)
I20250111 02:54:10.949162 6779 tablet_copy_service.cc:161] P bd3f4281bc0f4958b77f4304323bbd07: Beginning new tablet copy session on tablet 0741856ace474541b370b5579ab9093b from peer 47c9f617ebe940cd99b353e491c8d7be at {username='slave'} at 127.0.0.1:54304: session id = 47c9f617ebe940cd99b353e491c8d7be-0741856ace474541b370b5579ab9093b
I20250111 02:54:10.951591 6779 tablet_copy_source_session.cc:215] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:54:10.952841 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0741856ace474541b370b5579ab9093b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:10.957644 7009 tablet_copy_client.cc:806] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Starting download of 0 data blocks...
I20250111 02:54:10.957902 7009 tablet_copy_client.cc:670] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Starting download of 1 WAL segments...
I20250111 02:54:10.959512 7009 tablet_copy_client.cc:538] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:54:10.962428 7009 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap starting.
I20250111 02:54:10.968351 7009 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:10.968657 7009 tablet_bootstrap.cc:492] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: Bootstrap complete.
I20250111 02:54:10.968922 7009 ts_tablet_manager.cc:1397] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20250111 02:54:10.969632 7009 raft_consensus.cc:357] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } attrs { replace: true } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
I20250111 02:54:10.969902 7009 raft_consensus.cc:738] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 47c9f617ebe940cd99b353e491c8d7be, State: Initialized, Role: LEARNER
I20250111 02:54:10.970124 7009 consensus_queue.cc:260] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "bd3f4281bc0f4958b77f4304323bbd07" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 46417 } } peers { permanent_uuid: "f920acdfe48e48cb9bd1463983b819f1" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 43471 } } peers { permanent_uuid: "16ab852a34d94de69641565f2bbd2881" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 40961 } attrs { replace: true } } peers { permanent_uuid: "47c9f617ebe940cd99b353e491c8d7be" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 34069 } attrs { promote: true } }
I20250111 02:54:10.970688 7009 ts_tablet_manager.cc:1428] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20250111 02:54:10.971248 6779 tablet_copy_service.cc:342] P bd3f4281bc0f4958b77f4304323bbd07: Request end of tablet copy session 47c9f617ebe940cd99b353e491c8d7be-0741856ace474541b370b5579ab9093b received from {username='slave'} at 127.0.0.1:54304
I20250111 02:54:10.971356 6779 tablet_copy_service.cc:434] P bd3f4281bc0f4958b77f4304323bbd07: ending tablet copy session 47c9f617ebe940cd99b353e491c8d7be-0741856ace474541b370b5579ab9093b on tablet 0741856ace474541b370b5579ab9093b with peer 47c9f617ebe940cd99b353e491c8d7be
I20250111 02:54:11.416340 6674 raft_consensus.cc:1212] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250111 02:54:11.458156 6674 raft_consensus.cc:1212] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3] Dedup: 3.3->[]
I20250111 02:54:11.485831 6674 raft_consensus.cc:1212] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250111 02:54:11.652583 412 tablet_server.cc:178] TabletServer@127.0.103.1:46417 shutting down...
W20250111 02:54:11.656086 6349 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 14 similar messages]
W20250111 02:54:11.658669 6352 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250111 02:54:11.658789 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250111 02:54:11.661285 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:11.661823 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.662077 412 raft_consensus.cc:2238] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 3 LEADER]: Raft consensus shutting down.
I20250111 02:54:11.662446 412 raft_consensus.cc:2267] T 3f4cf57ca748467ead76333d1e58a20f P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Raft consensus is shut down!
W20250111 02:54:11.663177 6889 consensus_peers.cc:487] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 -> Peer bd3f4281bc0f4958b77f4304323bbd07 (127.0.103.1:46417): Couldn't send request to peer bd3f4281bc0f4958b77f4304323bbd07. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:11.663209 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.663353 412 raft_consensus.cc:2238] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 3 LEADER]: Raft consensus shutting down.
I20250111 02:54:11.663733 412 raft_consensus.cc:2267] T 0741856ace474541b370b5579ab9093b P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.664204 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.664328 412 raft_consensus.cc:2238] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:11.664491 412 raft_consensus.cc:2267] T 486aa5274c4b4f3aa94a5c82682d146f P bd3f4281bc0f4958b77f4304323bbd07 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.664954 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.665078 412 raft_consensus.cc:2238] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:11.665256 412 raft_consensus.cc:2267] T d9c4f53f3a2e4b49b9ef215ea5454686 P bd3f4281bc0f4958b77f4304323bbd07 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.680435 412 tablet_server.cc:195] TabletServer@127.0.103.1:46417 shutdown complete.
I20250111 02:54:11.684655 412 tablet_server.cc:178] TabletServer@127.0.103.2:43471 shutting down...
I20250111 02:54:11.693710 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:11.694011 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.694226 412 raft_consensus.cc:2238] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:11.694411 412 raft_consensus.cc:2267] T 3f4cf57ca748467ead76333d1e58a20f P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.695009 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.695176 412 raft_consensus.cc:2238] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:11.695369 412 raft_consensus.cc:2267] T 0741856ace474541b370b5579ab9093b P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.695883 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.696024 412 raft_consensus.cc:2238] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:11.696214 412 raft_consensus.cc:2267] T 486aa5274c4b4f3aa94a5c82682d146f P f920acdfe48e48cb9bd1463983b819f1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.696695 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.696825 412 raft_consensus.cc:2238] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:11.697003 412 raft_consensus.cc:2267] T d9c4f53f3a2e4b49b9ef215ea5454686 P f920acdfe48e48cb9bd1463983b819f1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.711478 412 tablet_server.cc:195] TabletServer@127.0.103.2:43471 shutdown complete.
I20250111 02:54:11.723523 412 tablet_server.cc:178] TabletServer@127.0.103.3:40961 shutting down...
W20250111 02:54:11.725646 6352 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250111 02:54:11.725759 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250111 02:54:11.727782 6352 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250111 02:54:11.727897 6352 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250111 02:54:11.732292 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:11.732498 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.732669 412 raft_consensus.cc:2238] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:11.732825 412 raft_consensus.cc:2267] T 3f4cf57ca748467ead76333d1e58a20f P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.733353 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.733484 412 raft_consensus.cc:2238] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:11.733634 412 raft_consensus.cc:2267] T 0741856ace474541b370b5579ab9093b P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.734077 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.734197 412 raft_consensus.cc:2238] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:54:11.734513 412 raft_consensus.cc:2267] T 486aa5274c4b4f3aa94a5c82682d146f P 16ab852a34d94de69641565f2bbd2881 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.734901 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:11.735037 412 raft_consensus.cc:2238] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 3 LEADER]: Raft consensus shutting down.
I20250111 02:54:11.735357 412 raft_consensus.cc:2267] T d9c4f53f3a2e4b49b9ef215ea5454686 P 16ab852a34d94de69641565f2bbd2881 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.749425 412 tablet_server.cc:195] TabletServer@127.0.103.3:40961 shutdown complete.
I20250111 02:54:11.752894 412 tablet_server.cc:178] TabletServer@127.0.103.4:0 shutting down...
I20250111 02:54:11.760725 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:11.760957 412 tablet_replica.cc:331] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be: stopping tablet replica
I20250111 02:54:11.761164 412 raft_consensus.cc:2238] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Raft consensus shutting down.
I20250111 02:54:11.761363 412 raft_consensus.cc:2267] T 0741856ace474541b370b5579ab9093b P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Raft consensus is shut down!
I20250111 02:54:11.761965 412 tablet_replica.cc:331] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be: stopping tablet replica
I20250111 02:54:11.762096 412 raft_consensus.cc:2238] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be [term 2 LEARNER]: Raft consensus shutting down.
I20250111 02:54:11.762226 412 raft_consensus.cc:2267] T 486aa5274c4b4f3aa94a5c82682d146f P 47c9f617ebe940cd99b353e491c8d7be [term 2 LEARNER]: Raft consensus is shut down!
I20250111 02:54:11.762655 412 tablet_replica.cc:331] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be: stopping tablet replica
I20250111 02:54:11.762766 412 raft_consensus.cc:2238] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Raft consensus shutting down.
I20250111 02:54:11.762878 412 raft_consensus.cc:2267] T d9c4f53f3a2e4b49b9ef215ea5454686 P 47c9f617ebe940cd99b353e491c8d7be [term 3 LEARNER]: Raft consensus is shut down!
I20250111 02:54:11.776295 412 tablet_server.cc:195] TabletServer@127.0.103.4:0 shutdown complete.
I20250111 02:54:11.779517 412 master.cc:537] Master@127.0.103.62:38351 shutting down...
I20250111 02:54:11.786957 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:11.787171 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:11.787263 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 375dab4837a34932b9d2fcb074d6b7b6: stopping tablet replica
I20250111 02:54:11.800165 412 master.cc:559] Master@127.0.103.62:38351 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11612 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20250111 02:54:11.813959 412 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.62:43731
I20250111 02:54:11.814462 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:11.816728 7026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:11.817943 7027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:11.818974 7029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:11.838325 412 server_base.cc:1034] running on GCE node
I20250111 02:54:11.838582 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:11.838630 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:11.838652 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564051838650 us; error 0 us; skew 500 ppm
I20250111 02:54:11.838768 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:11.839957 412 webserver.cc:458] Webserver started at http://127.0.103.62:34383/ using document root <none> and password file <none>
I20250111 02:54:11.840152 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:11.840205 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:11.840304 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:11.840875 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/master-0-root/instance:
uuid: "5c363f13bd8a4021b152ce6d3c94cc3d"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:11.843206 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:54:11.844633 7034 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:11.844929 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250111 02:54:11.845026 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/master-0-root
uuid: "5c363f13bd8a4021b152ce6d3c94cc3d"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:11.845125 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:11.855706 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:11.856187 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:11.866715 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.62:43731
I20250111 02:54:11.866744 7085 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.62:43731 every 8 connection(s)
I20250111 02:54:11.867888 7086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:11.871695 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: Bootstrap starting.
I20250111 02:54:11.872886 7086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:11.874256 7086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: No bootstrap required, opened a new log
I20250111 02:54:11.874821 7086 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c363f13bd8a4021b152ce6d3c94cc3d" member_type: VOTER }
I20250111 02:54:11.874945 7086 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:11.874996 7086 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c363f13bd8a4021b152ce6d3c94cc3d, State: Initialized, Role: FOLLOWER
I20250111 02:54:11.875185 7086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c363f13bd8a4021b152ce6d3c94cc3d" member_type: VOTER }
I20250111 02:54:11.875283 7086 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250111 02:54:11.875334 7086 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250111 02:54:11.875407 7086 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:11.877189 7086 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c363f13bd8a4021b152ce6d3c94cc3d" member_type: VOTER }
I20250111 02:54:11.877342 7086 leader_election.cc:304] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c363f13bd8a4021b152ce6d3c94cc3d; no voters:
I20250111 02:54:11.877771 7086 leader_election.cc:290] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250111 02:54:11.877923 7089 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:11.878335 7089 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 1 LEADER]: Becoming Leader. State: Replica: 5c363f13bd8a4021b152ce6d3c94cc3d, State: Running, Role: LEADER
I20250111 02:54:11.878543 7089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c363f13bd8a4021b152ce6d3c94cc3d" member_type: VOTER }
I20250111 02:54:11.878734 7086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [sys.catalog]: configured and running, proceeding with master startup.
I20250111 02:54:11.879297 7091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c363f13bd8a4021b152ce6d3c94cc3d. Latest consensus state: current_term: 1 leader_uuid: "5c363f13bd8a4021b152ce6d3c94cc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c363f13bd8a4021b152ce6d3c94cc3d" member_type: VOTER } }
I20250111 02:54:11.879329 7090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c363f13bd8a4021b152ce6d3c94cc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c363f13bd8a4021b152ce6d3c94cc3d" member_type: VOTER } }
I20250111 02:54:11.879460 7091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [sys.catalog]: This master's current role is: LEADER
I20250111 02:54:11.879617 7090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [sys.catalog]: This master's current role is: LEADER
I20250111 02:54:11.882208 7093 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250111 02:54:11.883551 7093 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250111 02:54:11.883889 412 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250111 02:54:11.886436 7093 catalog_manager.cc:1348] Generated new cluster ID: 90181b12b5404451964ea330c6901031
I20250111 02:54:11.886514 7093 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250111 02:54:11.903985 7093 catalog_manager.cc:1371] Generated new certificate authority record
I20250111 02:54:11.904780 7093 catalog_manager.cc:1505] Loading token signing keys...
I20250111 02:54:11.914197 7093 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: Generated new TSK 0
I20250111 02:54:11.914454 7093 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250111 02:54:11.916427 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:11.919095 7107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:11.920114 7108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:11.921241 7110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:11.922223 412 server_base.cc:1034] running on GCE node
I20250111 02:54:11.922479 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:11.922531 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:11.922554 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564051922551 us; error 0 us; skew 500 ppm
I20250111 02:54:11.922722 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:11.923959 412 webserver.cc:458] Webserver started at http://127.0.103.1:44679/ using document root <none> and password file <none>
I20250111 02:54:11.924167 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:11.924227 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:11.924392 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:11.925038 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root/instance:
uuid: "502ca883a5d74182a320d1d213c3d0c0"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:11.927515 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:54:11.928958 7115 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:11.929243 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:54:11.929339 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "502ca883a5d74182a320d1d213c3d0c0"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:11.929457 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:11.943549 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:11.944039 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:11.944556 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:11.945382 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:11.945441 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:11.945536 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:11.945571 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:11.957007 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:37843
I20250111 02:54:11.957041 7177 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:37843 every 8 connection(s)
I20250111 02:54:11.958609 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:11.961457 7182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:11.962482 7183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:11.963415 7185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:11.963791 412 server_base.cc:1034] running on GCE node
I20250111 02:54:11.964010 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:11.964066 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:11.964109 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564051964105 us; error 0 us; skew 500 ppm
I20250111 02:54:11.964246 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:11.965224 7178 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43731
I20250111 02:54:11.965350 7178 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:11.965641 412 webserver.cc:458] Webserver started at http://127.0.103.2:40501/ using document root <none> and password file <none>
I20250111 02:54:11.965772 7178 heartbeater.cc:510] Master 127.0.103.62:43731 requested a full tablet report, sending...
I20250111 02:54:11.965898 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:11.965971 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:11.966085 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:11.966635 7051 ts_manager.cc:194] Registered new tserver with Master: 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843)
I20250111 02:54:11.966683 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root/instance:
uuid: "ec1ece0a31324840be298ddc7e853d1b"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:11.967622 7051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35740
I20250111 02:54:11.969193 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250111 02:54:11.970610 7190 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:11.970883 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:54:11.970963 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root
uuid: "ec1ece0a31324840be298ddc7e853d1b"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:11.971061 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:11.982077 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:11.982502 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:11.982964 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:11.983726 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:11.983811 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:11.983901 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:11.983952 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:11.994993 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.2:38955
I20250111 02:54:11.995009 7252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.2:38955 every 8 connection(s)
I20250111 02:54:11.996546 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:11.999270 7256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:12.000389 7257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:12.000710 412 server_base.cc:1034] running on GCE node
W20250111 02:54:12.001492 7259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:12.001839 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:12.001945 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:12.001982 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564052001979 us; error 0 us; skew 500 ppm
I20250111 02:54:12.002087 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:12.002421 7253 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43731
I20250111 02:54:12.002542 7253 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:12.002924 7253 heartbeater.cc:510] Master 127.0.103.62:43731 requested a full tablet report, sending...
I20250111 02:54:12.003472 412 webserver.cc:458] Webserver started at http://127.0.103.3:41421/ using document root <none> and password file <none>
I20250111 02:54:12.003705 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:12.003772 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:12.003822 7051 ts_manager.cc:194] Registered new tserver with Master: ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:12.003883 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:12.004477 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root/instance:
uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:12.004577 7051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35752
I20250111 02:54:12.007205 412 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250111 02:54:12.008550 7264 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:12.008824 412 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250111 02:54:12.008919 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root
uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:12.009009 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:12.025169 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:12.025645 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:12.026226 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:12.027012 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:12.027079 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:12.027144 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:12.027177 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:12.038692 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.3:37345
I20250111 02:54:12.038728 7326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.3:37345 every 8 connection(s)
I20250111 02:54:12.044775 7327 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43731
I20250111 02:54:12.044901 7327 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:12.045218 7327 heartbeater.cc:510] Master 127.0.103.62:43731 requested a full tablet report, sending...
I20250111 02:54:12.046046 7051 ts_manager.cc:194] Registered new tserver with Master: ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:12.046221 412 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006721066s
I20250111 02:54:12.046808 7051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35754
I20250111 02:54:12.969007 7178 heartbeater.cc:502] Master 127.0.103.62:43731 was elected leader, sending a full tablet report...
I20250111 02:54:13.006093 7253 heartbeater.cc:502] Master 127.0.103.62:43731 was elected leader, sending a full tablet report...
I20250111 02:54:13.048067 7327 heartbeater.cc:502] Master 127.0.103.62:43731 was elected leader, sending a full tablet report...
I20250111 02:54:13.073989 412 test_util.cc:274] Using random seed: 1401118782
I20250111 02:54:13.092707 7051 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35766:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:54:13.093392 7051 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:54:13.125747 7140 tablet_service.cc:1467] Processing CreateTablet for tablet a0c1d0c792c4443c962a2a013e49e3af (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:13.126144 7142 tablet_service.cc:1467] Processing CreateTablet for tablet 85776db200cd43268074d11d9c659f79 (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:13.126363 7140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0c1d0c792c4443c962a2a013e49e3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.126559 7142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85776db200cd43268074d11d9c659f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.128846 7289 tablet_service.cc:1467] Processing CreateTablet for tablet a0c1d0c792c4443c962a2a013e49e3af (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:13.129272 7289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0c1d0c792c4443c962a2a013e49e3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.129227 7141 tablet_service.cc:1467] Processing CreateTablet for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:13.129608 7141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ccdf9e9bdf46179da1c099ea14c3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.133565 7290 tablet_service.cc:1467] Processing CreateTablet for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:13.134140 7291 tablet_service.cc:1467] Processing CreateTablet for tablet 85776db200cd43268074d11d9c659f79 (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:13.134522 7291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85776db200cd43268074d11d9c659f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.136155 7292 tablet_service.cc:1467] Processing CreateTablet for tablet bd66331e08224e0a9b7b0db16d52e8aa (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:13.136482 7292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd66331e08224e0a9b7b0db16d52e8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.138870 7218 tablet_service.cc:1467] Processing CreateTablet for tablet bd66331e08224e0a9b7b0db16d52e8aa (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:13.139356 7218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd66331e08224e0a9b7b0db16d52e8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.139689 7348 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b: Bootstrap starting.
I20250111 02:54:13.125748 7143 tablet_service.cc:1467] Processing CreateTablet for tablet bd66331e08224e0a9b7b0db16d52e8aa (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:13.142331 7348 tablet_bootstrap.cc:654] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.142380 7143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd66331e08224e0a9b7b0db16d52e8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.142995 7290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ccdf9e9bdf46179da1c099ea14c3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.144688 7347 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:13.146435 7217 tablet_service.cc:1467] Processing CreateTablet for tablet 85776db200cd43268074d11d9c659f79 (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:13.146853 7217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85776db200cd43268074d11d9c659f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.147127 7215 tablet_service.cc:1467] Processing CreateTablet for tablet a0c1d0c792c4443c962a2a013e49e3af (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:13.147516 7215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0c1d0c792c4443c962a2a013e49e3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.146476 7216 tablet_service.cc:1467] Processing CreateTablet for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 (DEFAULT_TABLE table=test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:13.150468 7216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ccdf9e9bdf46179da1c099ea14c3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.151284 7348 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b: No bootstrap required, opened a new log
I20250111 02:54:13.151458 7348 ts_tablet_manager.cc:1397] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.002s
I20250111 02:54:13.154559 7347 tablet_bootstrap.cc:654] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.152496 7348 raft_consensus.cc:357] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.155587 7348 raft_consensus.cc:383] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.155685 7348 raft_consensus.cc:738] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.155973 7348 consensus_queue.cc:260] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.156978 7350 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b: Bootstrap starting.
I20250111 02:54:13.157466 7348 ts_tablet_manager.cc:1428] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20250111 02:54:13.157794 7348 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b: Bootstrap starting.
I20250111 02:54:13.158524 7347 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: No bootstrap required, opened a new log
I20250111 02:54:13.158685 7347 ts_tablet_manager.cc:1397] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250111 02:54:13.159125 7350 tablet_bootstrap.cc:654] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.159430 7347 raft_consensus.cc:357] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.159729 7348 tablet_bootstrap.cc:654] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.159729 7347 raft_consensus.cc:383] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.159925 7347 raft_consensus.cc:738] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.160168 7347 consensus_queue.cc:260] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.162987 7347 ts_tablet_manager.cc:1428] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250111 02:54:13.163376 7350 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b: No bootstrap required, opened a new log
I20250111 02:54:13.163508 7350 ts_tablet_manager.cc:1397] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20250111 02:54:13.163686 7348 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b: No bootstrap required, opened a new log
I20250111 02:54:13.163782 7347 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:13.163890 7348 ts_tablet_manager.cc:1397] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250111 02:54:13.164680 7348 raft_consensus.cc:357] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.164930 7348 raft_consensus.cc:383] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.165032 7348 raft_consensus.cc:738] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.165261 7347 tablet_bootstrap.cc:654] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.165381 7348 consensus_queue.cc:260] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.165701 7350 raft_consensus.cc:357] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.166091 7348 ts_tablet_manager.cc:1428] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250111 02:54:13.166657 7350 raft_consensus.cc:383] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.166795 7350 raft_consensus.cc:738] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.167033 7348 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: Bootstrap starting.
I20250111 02:54:13.166976 7350 consensus_queue.cc:260] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.168888 7348 tablet_bootstrap.cc:654] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.171150 7348 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: No bootstrap required, opened a new log
I20250111 02:54:13.171384 7348 ts_tablet_manager.cc:1397] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20250111 02:54:13.171638 7355 raft_consensus.cc:491] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.171798 7355 raft_consensus.cc:513] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.172952 7355 leader_election.cc:290] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.174563 7347 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: No bootstrap required, opened a new log
I20250111 02:54:13.174736 7347 ts_tablet_manager.cc:1397] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20250111 02:54:13.175709 7347 raft_consensus.cc:357] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.175963 7347 raft_consensus.cc:383] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.176036 7347 raft_consensus.cc:738] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.176296 7347 consensus_queue.cc:260] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.176900 7347 ts_tablet_manager.cc:1428] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250111 02:54:13.180335 7350 ts_tablet_manager.cc:1428] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b: Time spent starting tablet: real 0.017s user 0.001s sys 0.009s
I20250111 02:54:13.180871 7350 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b: Bootstrap starting.
I20250111 02:54:13.180474 7348 raft_consensus.cc:357] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.181180 7348 raft_consensus.cc:383] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.181252 7348 raft_consensus.cc:738] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.181711 7348 consensus_queue.cc:260] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.182870 7350 tablet_bootstrap.cc:654] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.186439 7348 ts_tablet_manager.cc:1428] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20250111 02:54:13.186822 7348 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: Bootstrap starting.
I20250111 02:54:13.186874 7347 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:13.188395 7348 tablet_bootstrap.cc:654] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.188962 7347 tablet_bootstrap.cc:654] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.193374 7350 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b: No bootstrap required, opened a new log
I20250111 02:54:13.193699 7350 ts_tablet_manager.cc:1397] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250111 02:54:13.194572 7350 raft_consensus.cc:357] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.194785 7350 raft_consensus.cc:383] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.194864 7350 raft_consensus.cc:738] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.195111 7350 consensus_queue.cc:260] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.195616 7350 ts_tablet_manager.cc:1428] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250111 02:54:13.199270 7347 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: No bootstrap required, opened a new log
I20250111 02:54:13.199651 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b" is_pre_election: true
I20250111 02:54:13.199468 7347 ts_tablet_manager.cc:1397] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
W20250111 02:54:13.200281 7117 leader_election.cc:343] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:54:13.200557 7348 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: No bootstrap required, opened a new log
I20250111 02:54:13.200706 7348 ts_tablet_manager.cc:1397] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20250111 02:54:13.200706 7350 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: Bootstrap starting.
I20250111 02:54:13.200879 7347 raft_consensus.cc:357] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.201066 7347 raft_consensus.cc:383] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.201169 7347 raft_consensus.cc:738] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.201449 7348 raft_consensus.cc:357] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.201411 7347 consensus_queue.cc:260] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.201591 7348 raft_consensus.cc:383] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.201687 7348 raft_consensus.cc:738] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.201920 7348 consensus_queue.cc:260] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.202075 7347 ts_tablet_manager.cc:1428] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250111 02:54:13.202304 7350 tablet_bootstrap.cc:654] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.202544 7347 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:13.203821 7348 ts_tablet_manager.cc:1428] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:54:13.204946 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
I20250111 02:54:13.205137 7302 raft_consensus.cc:2463] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 502ca883a5d74182a320d1d213c3d0c0 in term 0.
I20250111 02:54:13.205514 7118 leader_election.cc:304] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b; no voters: ec1ece0a31324840be298ddc7e853d1b
I20250111 02:54:13.205788 7355 raft_consensus.cc:2798] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:13.205910 7355 raft_consensus.cc:491] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:13.206004 7355 raft_consensus.cc:3054] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.206085 7347 tablet_bootstrap.cc:654] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.209221 7355 raft_consensus.cc:513] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.210425 7355 leader_election.cc:290] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 election: Requested vote from peers ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.210387 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
I20250111 02:54:13.210636 7302 raft_consensus.cc:3054] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.210736 7350 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: No bootstrap required, opened a new log
I20250111 02:54:13.210865 7350 ts_tablet_manager.cc:1397] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20250111 02:54:13.211503 7350 raft_consensus.cc:357] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.211588 7347 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: No bootstrap required, opened a new log
I20250111 02:54:13.211625 7350 raft_consensus.cc:383] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.211659 7350 raft_consensus.cc:738] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.211690 7347 ts_tablet_manager.cc:1397] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20250111 02:54:13.211822 7350 consensus_queue.cc:260] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.212225 7350 ts_tablet_manager.cc:1428] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:54:13.212319 7347 raft_consensus.cc:357] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.212483 7350 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b: Bootstrap starting.
I20250111 02:54:13.212482 7347 raft_consensus.cc:383] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.212586 7347 raft_consensus.cc:738] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.212749 7347 consensus_queue.cc:260] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.213212 7347 ts_tablet_manager.cc:1428] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250111 02:54:13.213274 7302 raft_consensus.cc:2463] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 502ca883a5d74182a320d1d213c3d0c0 in term 1.
I20250111 02:54:13.213536 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b"
I20250111 02:54:13.213622 7118 leader_election.cc:304] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b; no voters:
I20250111 02:54:13.213753 7350 tablet_bootstrap.cc:654] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.213892 7355 raft_consensus.cc:2798] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader election won for term 1
W20250111 02:54:13.214740 7117 leader_election.cc:343] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:54:13.217684 7355 raft_consensus.cc:695] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 LEADER]: Becoming Leader. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Running, Role: LEADER
I20250111 02:54:13.217983 7355 consensus_queue.cc:237] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.218164 7350 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b: No bootstrap required, opened a new log
I20250111 02:54:13.218289 7350 ts_tablet_manager.cc:1397] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250111 02:54:13.218946 7350 raft_consensus.cc:357] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.219167 7350 raft_consensus.cc:383] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.219238 7350 raft_consensus.cc:738] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.220119 7350 consensus_queue.cc:260] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.220611 7350 ts_tablet_manager.cc:1428] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:13.220533 7051 catalog_manager.cc:5526] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "502ca883a5d74182a320d1d213c3d0c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:13.253242 7355 raft_consensus.cc:491] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.253448 7355 raft_consensus.cc:513] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.254132 7355 leader_election.cc:290] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:13.254609 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
I20250111 02:54:13.254609 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b" is_pre_election: true
I20250111 02:54:13.254838 7302 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 502ca883a5d74182a320d1d213c3d0c0 in term 0.
I20250111 02:54:13.254838 7228 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 502ca883a5d74182a320d1d213c3d0c0 in term 0.
I20250111 02:54:13.255214 7117 leader_election.cc:304] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.255429 7355 raft_consensus.cc:2798] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:13.255498 7355 raft_consensus.cc:491] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:13.255544 7355 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.257418 7355 raft_consensus.cc:513] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.257989 7355 leader_election.cc:290] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 election: Requested vote from peers ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:13.258407 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
I20250111 02:54:13.258441 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b"
I20250111 02:54:13.258558 7302 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.258611 7228 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.260416 7228 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 502ca883a5d74182a320d1d213c3d0c0 in term 1.
I20250111 02:54:13.260425 7302 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 502ca883a5d74182a320d1d213c3d0c0 in term 1.
I20250111 02:54:13.260787 7118 leader_election.cc:304] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b; no voters:
I20250111 02:54:13.260996 7355 raft_consensus.cc:2798] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:13.261101 7355 raft_consensus.cc:695] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 LEADER]: Becoming Leader. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Running, Role: LEADER
I20250111 02:54:13.261368 7355 consensus_queue.cc:237] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:13.263846 7051 catalog_manager.cc:5526] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1). New cstate: current_term: 1 leader_uuid: "502ca883a5d74182a320d1d213c3d0c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:13.275696 7352 raft_consensus.cc:491] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.275857 7352 raft_consensus.cc:513] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.276575 7352 leader_election.cc:290] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:13.282122 7357 raft_consensus.cc:491] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.282305 7357 raft_consensus.cc:513] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.283308 7357 leader_election.cc:290] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.283535 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c1d0c792c4443c962a2a013e49e3af" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b" is_pre_election: true
I20250111 02:54:13.283743 7228 raft_consensus.cc:2463] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 0.
I20250111 02:54:13.283838 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c1d0c792c4443c962a2a013e49e3af" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:13.284039 7153 raft_consensus.cc:2463] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 0.
I20250111 02:54:13.284602 7268 leader_election.cc:304] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b; no voters:
I20250111 02:54:13.284912 7352 raft_consensus.cc:2798] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:13.284988 7352 raft_consensus.cc:491] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:13.285064 7352 raft_consensus.cc:3054] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.287967 7352 raft_consensus.cc:513] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.288575 7352 leader_election.cc:290] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:13.289171 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c1d0c792c4443c962a2a013e49e3af" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b"
I20250111 02:54:13.289171 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c1d0c792c4443c962a2a013e49e3af" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:13.289404 7228 raft_consensus.cc:3054] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.289415 7153 raft_consensus.cc:3054] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.290993 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c1d0c792c4443c962a2a013e49e3af" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
I20250111 02:54:13.291211 7302 raft_consensus.cc:2388] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec1ece0a31324840be298ddc7e853d1b in current term 1: Already voted for candidate ddfa5c4f3c534b3b96b0269100068a4b in this term.
I20250111 02:54:13.291990 7228 raft_consensus.cc:2463] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 1.
I20250111 02:54:13.291990 7153 raft_consensus.cc:2463] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 1.
I20250111 02:54:13.292300 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0c1d0c792c4443c962a2a013e49e3af" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:13.292498 7152 raft_consensus.cc:2388] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec1ece0a31324840be298ddc7e853d1b in current term 1: Already voted for candidate ddfa5c4f3c534b3b96b0269100068a4b in this term.
I20250111 02:54:13.292454 7266 leader_election.cc:304] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddfa5c4f3c534b3b96b0269100068a4b, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.292680 7352 raft_consensus.cc:2798] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:13.292989 7194 leader_election.cc:304] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec1ece0a31324840be298ddc7e853d1b; no voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b
I20250111 02:54:13.293259 7352 raft_consensus.cc:695] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Becoming Leader. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Running, Role: LEADER
I20250111 02:54:13.293305 7357 raft_consensus.cc:2743] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:54:13.293519 7352 consensus_queue.cc:237] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.295812 7051 catalog_manager.cc:5526] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: term changed from 0 to 1, leader changed from <none> to ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3). New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } }
I20250111 02:54:13.370450 7352 raft_consensus.cc:491] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.370640 7352 raft_consensus.cc:513] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.371294 7352 leader_election.cc:290] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:13.371807 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ccdf9e9bdf46179da1c099ea14c3b6" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:13.371852 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ccdf9e9bdf46179da1c099ea14c3b6" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b" is_pre_election: true
I20250111 02:54:13.372001 7153 raft_consensus.cc:2463] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 0.
I20250111 02:54:13.372023 7228 raft_consensus.cc:2463] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 0.
I20250111 02:54:13.372380 7266 leader_election.cc:304] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddfa5c4f3c534b3b96b0269100068a4b, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.372629 7352 raft_consensus.cc:2798] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:13.372695 7352 raft_consensus.cc:491] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:13.372754 7352 raft_consensus.cc:3054] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.374606 7352 raft_consensus.cc:513] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.375198 7352 leader_election.cc:290] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:13.375550 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ccdf9e9bdf46179da1c099ea14c3b6" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:13.375659 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ccdf9e9bdf46179da1c099ea14c3b6" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b"
I20250111 02:54:13.375725 7153 raft_consensus.cc:3054] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.375851 7228 raft_consensus.cc:3054] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.377643 7153 raft_consensus.cc:2463] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 1.
I20250111 02:54:13.377645 7228 raft_consensus.cc:2463] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 1.
I20250111 02:54:13.378002 7268 leader_election.cc:304] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.378234 7352 raft_consensus.cc:2798] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:13.378360 7352 raft_consensus.cc:695] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Becoming Leader. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Running, Role: LEADER
I20250111 02:54:13.378607 7352 consensus_queue.cc:237] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.381001 7051 catalog_manager.cc:5526] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: term changed from 0 to 1, leader changed from <none> to ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3). New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } }
I20250111 02:54:13.432588 412 test_util.cc:274] Using random seed: 1401477374
I20250111 02:54:13.443881 7051 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35780:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250111 02:54:13.444531 7051 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250111 02:54:13.451951 7142 tablet_service.cc:1467] Processing CreateTablet for tablet 10c7b479b15c4335912eb49e40aec3d0 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:13.452441 7142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c7b479b15c4335912eb49e40aec3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.452535 7143 tablet_service.cc:1467] Processing CreateTablet for tablet b0ee7290ae1847c390db93c34bef48bf (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:13.452905 7143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0ee7290ae1847c390db93c34bef48bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.453785 7216 tablet_service.cc:1467] Processing CreateTablet for tablet 10c7b479b15c4335912eb49e40aec3d0 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:13.454169 7216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c7b479b15c4335912eb49e40aec3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.457289 7347 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:13.458250 7141 tablet_service.cc:1467] Processing CreateTablet for tablet f664c6e13c3a46ea927f0e9713ef6b91 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:13.458359 7289 tablet_service.cc:1467] Processing CreateTablet for tablet 10c7b479b15c4335912eb49e40aec3d0 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250111 02:54:13.458631 7141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f664c6e13c3a46ea927f0e9713ef6b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.458714 7289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10c7b479b15c4335912eb49e40aec3d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.458767 7347 tablet_bootstrap.cc:654] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.459285 7140 tablet_service.cc:1467] Processing CreateTablet for tablet d5974446b8044cb5b2146db5ceb7c5a3 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:13.459599 7140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5974446b8044cb5b2146db5ceb7c5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.463888 7218 tablet_service.cc:1467] Processing CreateTablet for tablet b0ee7290ae1847c390db93c34bef48bf (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:13.464390 7218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0ee7290ae1847c390db93c34bef48bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.466182 7215 tablet_service.cc:1467] Processing CreateTablet for tablet d5974446b8044cb5b2146db5ceb7c5a3 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:13.466545 7215 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5974446b8044cb5b2146db5ceb7c5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.466755 7348 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: Bootstrap starting.
I20250111 02:54:13.466789 7347 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: No bootstrap required, opened a new log
I20250111 02:54:13.466892 7347 ts_tablet_manager.cc:1397] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20250111 02:54:13.467697 7347 raft_consensus.cc:357] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.467922 7347 raft_consensus.cc:383] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.467985 7347 raft_consensus.cc:738] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.468292 7348 tablet_bootstrap.cc:654] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.468243 7347 consensus_queue.cc:260] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.468906 7347 ts_tablet_manager.cc:1428] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250111 02:54:13.469238 7347 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:13.470218 7217 tablet_service.cc:1467] Processing CreateTablet for tablet f664c6e13c3a46ea927f0e9713ef6b91 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:13.470583 7217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f664c6e13c3a46ea927f0e9713ef6b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.470615 7350 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: Bootstrap starting.
I20250111 02:54:13.471033 7347 tablet_bootstrap.cc:654] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.472016 7350 tablet_bootstrap.cc:654] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.473661 7290 tablet_service.cc:1467] Processing CreateTablet for tablet f664c6e13c3a46ea927f0e9713ef6b91 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250111 02:54:13.474211 7290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f664c6e13c3a46ea927f0e9713ef6b91. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.477769 7292 tablet_service.cc:1467] Processing CreateTablet for tablet d5974446b8044cb5b2146db5ceb7c5a3 (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250111 02:54:13.478186 7292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5974446b8044cb5b2146db5ceb7c5a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.478768 7347 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: No bootstrap required, opened a new log
I20250111 02:54:13.478940 7347 ts_tablet_manager.cc:1397] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20250111 02:54:13.479768 7347 raft_consensus.cc:357] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.480067 7347 raft_consensus.cc:383] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.480185 7347 raft_consensus.cc:738] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.480782 7347 consensus_queue.cc:260] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.481537 7347 ts_tablet_manager.cc:1428] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:13.480558 7289 tablet_service.cc:1467] Processing CreateTablet for tablet b0ee7290ae1847c390db93c34bef48bf (DEFAULT_TABLE table=dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250111 02:54:13.482241 7289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0ee7290ae1847c390db93c34bef48bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:13.482285 7347 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:13.483017 7350 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: No bootstrap required, opened a new log
I20250111 02:54:13.483202 7350 ts_tablet_manager.cc:1397] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20250111 02:54:13.484052 7350 raft_consensus.cc:357] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.484302 7350 raft_consensus.cc:383] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.484452 7350 raft_consensus.cc:738] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.484819 7350 consensus_queue.cc:260] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.485509 7350 ts_tablet_manager.cc:1428] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250111 02:54:13.485826 7350 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: Bootstrap starting.
I20250111 02:54:13.487551 7350 tablet_bootstrap.cc:654] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.488237 7348 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: No bootstrap required, opened a new log
I20250111 02:54:13.484395 7347 tablet_bootstrap.cc:654] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.488397 7348 ts_tablet_manager.cc:1397] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.002s
I20250111 02:54:13.489219 7348 raft_consensus.cc:357] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.489470 7348 raft_consensus.cc:383] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.489576 7348 raft_consensus.cc:738] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.489833 7348 consensus_queue.cc:260] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.490399 7348 ts_tablet_manager.cc:1428] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250111 02:54:13.490689 7348 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: Bootstrap starting.
I20250111 02:54:13.491719 7350 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: No bootstrap required, opened a new log
I20250111 02:54:13.491860 7350 ts_tablet_manager.cc:1397] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250111 02:54:13.491973 7347 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: No bootstrap required, opened a new log
I20250111 02:54:13.492143 7347 ts_tablet_manager.cc:1397] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20250111 02:54:13.492187 7348 tablet_bootstrap.cc:654] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.492625 7350 raft_consensus.cc:357] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.492835 7350 raft_consensus.cc:383] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.492921 7350 raft_consensus.cc:738] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.492873 7347 raft_consensus.cc:357] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.493078 7347 raft_consensus.cc:383] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.493147 7350 consensus_queue.cc:260] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.493163 7347 raft_consensus.cc:738] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.493700 7350 ts_tablet_manager.cc:1428] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250111 02:54:13.494055 7350 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: Bootstrap starting.
I20250111 02:54:13.495113 7348 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: No bootstrap required, opened a new log
I20250111 02:54:13.495260 7348 ts_tablet_manager.cc:1397] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250111 02:54:13.495545 7350 tablet_bootstrap.cc:654] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.495616 7347 consensus_queue.cc:260] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.496084 7348 raft_consensus.cc:357] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.496352 7348 raft_consensus.cc:383] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.496483 7348 raft_consensus.cc:738] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.496869 7348 consensus_queue.cc:260] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.497558 7347 ts_tablet_manager.cc:1428] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250111 02:54:13.497898 7347 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:13.498113 7348 ts_tablet_manager.cc:1428] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250111 02:54:13.498414 7348 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: Bootstrap starting.
I20250111 02:54:13.499748 7347 tablet_bootstrap.cc:654] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.500386 7348 tablet_bootstrap.cc:654] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.501210 7357 raft_consensus.cc:491] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.501397 7357 raft_consensus.cc:513] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.502239 7357 leader_election.cc:290] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.503033 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5974446b8044cb5b2146db5ceb7c5a3" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:13.503279 7153 raft_consensus.cc:2463] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 0.
I20250111 02:54:13.503711 7194 leader_election.cc:304] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.504011 7357 raft_consensus.cc:2798] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:13.504119 7357 raft_consensus.cc:491] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:13.504242 7357 raft_consensus.cc:3054] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.504668 7350 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: No bootstrap required, opened a new log
I20250111 02:54:13.504798 7350 ts_tablet_manager.cc:1397] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.002s
I20250111 02:54:13.505518 7350 raft_consensus.cc:357] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.505714 7350 raft_consensus.cc:383] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.505788 7350 raft_consensus.cc:738] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.506263 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5974446b8044cb5b2146db5ceb7c5a3" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
I20250111 02:54:13.507398 7357 raft_consensus.cc:513] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
W20250111 02:54:13.507580 7193 leader_election.cc:343] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:54:13.508152 7357 leader_election.cc:290] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.508347 7357 raft_consensus.cc:491] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.508499 7357 raft_consensus.cc:513] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.509230 7357 leader_election.cc:290] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.510179 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:13.510860 7348 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: No bootstrap required, opened a new log
I20250111 02:54:13.510773 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5974446b8044cb5b2146db5ceb7c5a3" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:13.511111 7153 raft_consensus.cc:3054] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.511770 7302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5974446b8044cb5b2146db5ceb7c5a3" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
I20250111 02:54:13.511940 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
W20250111 02:54:13.512203 7193 leader_election.cc:343] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345): Illegal state: must be running to vote when last-logged opid is not known
I20250111 02:54:13.512296 7301 raft_consensus.cc:2388] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec1ece0a31324840be298ddc7e853d1b in current term 1: Already voted for candidate 502ca883a5d74182a320d1d213c3d0c0 in this term.
I20250111 02:54:13.512729 7193 leader_election.cc:304] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec1ece0a31324840be298ddc7e853d1b; no voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b
I20250111 02:54:13.513103 7357 raft_consensus.cc:3054] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.507617 7350 consensus_queue.cc:260] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.513955 7153 raft_consensus.cc:2463] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 1.
I20250111 02:54:13.514380 7350 ts_tablet_manager.cc:1428] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250111 02:54:13.514520 7194 leader_election.cc:304] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters: ddfa5c4f3c534b3b96b0269100068a4b
I20250111 02:54:13.516319 7357 raft_consensus.cc:2743] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:54:13.517585 7350 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: Bootstrap starting.
I20250111 02:54:13.519734 7348 ts_tablet_manager.cc:1397] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20250111 02:54:13.520602 7357 raft_consensus.cc:2798] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:13.520988 7350 tablet_bootstrap.cc:654] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.521071 7357 raft_consensus.cc:695] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 1 LEADER]: Becoming Leader. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Running, Role: LEADER
I20250111 02:54:13.521431 7357 consensus_queue.cc:237] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.520607 7348 raft_consensus.cc:357] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.522787 7348 raft_consensus.cc:383] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.522892 7348 raft_consensus.cc:738] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.523154 7348 consensus_queue.cc:260] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.524780 7348 ts_tablet_manager.cc:1428] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250111 02:54:13.525157 7348 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: Bootstrap starting.
I20250111 02:54:13.527105 7348 tablet_bootstrap.cc:654] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: Neither blocks nor log segments found. Creating new log.
I20250111 02:54:13.529654 7050 catalog_manager.cc:5526] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec1ece0a31324840be298ddc7e853d1b (127.0.103.2). New cstate: current_term: 1 leader_uuid: "ec1ece0a31324840be298ddc7e853d1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:13.531045 7348 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: No bootstrap required, opened a new log
I20250111 02:54:13.531231 7348 ts_tablet_manager.cc:1397] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250111 02:54:13.531731 7347 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: No bootstrap required, opened a new log
I20250111 02:54:13.531941 7347 ts_tablet_manager.cc:1397] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.034s user 0.017s sys 0.015s
I20250111 02:54:13.531885 7348 raft_consensus.cc:357] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.532145 7348 raft_consensus.cc:383] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.532271 7348 raft_consensus.cc:738] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.532367 7350 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: No bootstrap required, opened a new log
I20250111 02:54:13.532567 7348 consensus_queue.cc:260] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.532739 7350 ts_tablet_manager.cc:1397] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20250111 02:54:13.532745 7347 raft_consensus.cc:357] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.533077 7347 raft_consensus.cc:383] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.533118 7348 ts_tablet_manager.cc:1428] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:13.533193 7347 raft_consensus.cc:738] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.533486 7347 consensus_queue.cc:260] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.533808 7350 raft_consensus.cc:357] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.534073 7350 raft_consensus.cc:383] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250111 02:54:13.534196 7350 raft_consensus.cc:738] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Initialized, Role: FOLLOWER
I20250111 02:54:13.534471 7350 consensus_queue.cc:260] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.534713 7347 ts_tablet_manager.cc:1428] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250111 02:54:13.535542 7350 ts_tablet_manager.cc:1428] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250111 02:54:13.546809 7357 raft_consensus.cc:491] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.546983 7357 raft_consensus.cc:513] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.547586 7357 leader_election.cc:290] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.548058 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f664c6e13c3a46ea927f0e9713ef6b91" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:13.548075 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f664c6e13c3a46ea927f0e9713ef6b91" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
I20250111 02:54:13.548274 7301 raft_consensus.cc:2463] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 0.
I20250111 02:54:13.548280 7153 raft_consensus.cc:2463] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 0.
I20250111 02:54:13.548657 7193 leader_election.cc:304] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddfa5c4f3c534b3b96b0269100068a4b, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.548897 7357 raft_consensus.cc:2798] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:13.548990 7357 raft_consensus.cc:491] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:13.549062 7357 raft_consensus.cc:3054] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.550887 7357 raft_consensus.cc:513] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.551492 7357 leader_election.cc:290] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.551985 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f664c6e13c3a46ea927f0e9713ef6b91" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:13.551982 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f664c6e13c3a46ea927f0e9713ef6b91" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
I20250111 02:54:13.552237 7301 raft_consensus.cc:3054] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.552244 7153 raft_consensus.cc:3054] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.554311 7153 raft_consensus.cc:2463] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 1.
I20250111 02:54:13.554317 7301 raft_consensus.cc:2463] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 1.
I20250111 02:54:13.554672 7194 leader_election.cc:304] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.554931 7357 raft_consensus.cc:2798] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:13.555051 7357 raft_consensus.cc:695] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 1 LEADER]: Becoming Leader. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Running, Role: LEADER
I20250111 02:54:13.555330 7357 consensus_queue.cc:237] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.557981 7051 catalog_manager.cc:5526] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec1ece0a31324840be298ddc7e853d1b (127.0.103.2). New cstate: current_term: 1 leader_uuid: "ec1ece0a31324840be298ddc7e853d1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:13.622134 7357 raft_consensus.cc:491] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.622371 7357 raft_consensus.cc:513] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.623098 7357 leader_election.cc:290] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.623638 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c7b479b15c4335912eb49e40aec3d0" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:13.623643 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c7b479b15c4335912eb49e40aec3d0" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
I20250111 02:54:13.623840 7153 raft_consensus.cc:2463] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 0.
I20250111 02:54:13.623948 7301 raft_consensus.cc:2463] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 0.
I20250111 02:54:13.624269 7194 leader_election.cc:304] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.624595 7357 raft_consensus.cc:2798] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:13.624675 7357 raft_consensus.cc:491] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:13.624785 7357 raft_consensus.cc:3054] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.626936 7357 raft_consensus.cc:513] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.627501 7357 leader_election.cc:290] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.627877 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c7b479b15c4335912eb49e40aec3d0" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:13.627934 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10c7b479b15c4335912eb49e40aec3d0" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
I20250111 02:54:13.628059 7153 raft_consensus.cc:3054] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.628120 7301 raft_consensus.cc:3054] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.629921 7301 raft_consensus.cc:2463] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 1.
I20250111 02:54:13.629968 7153 raft_consensus.cc:2463] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 1.
I20250111 02:54:13.630318 7194 leader_election.cc:304] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.630628 7357 raft_consensus.cc:2798] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:13.630790 7357 raft_consensus.cc:695] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 1 LEADER]: Becoming Leader. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Running, Role: LEADER
I20250111 02:54:13.631042 7357 consensus_queue.cc:237] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.633513 7051 catalog_manager.cc:5526] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec1ece0a31324840be298ddc7e853d1b (127.0.103.2). New cstate: current_term: 1 leader_uuid: "ec1ece0a31324840be298ddc7e853d1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:13.659574 7357 raft_consensus.cc:491] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.659821 7357 raft_consensus.cc:513] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.660494 7357 leader_election.cc:290] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.660903 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ee7290ae1847c390db93c34bef48bf" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:13.660975 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ee7290ae1847c390db93c34bef48bf" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
I20250111 02:54:13.661149 7153 raft_consensus.cc:2463] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 0.
I20250111 02:54:13.661108 7355 consensus_queue.cc:1035] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:13.661214 7301 raft_consensus.cc:2463] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 0.
I20250111 02:54:13.661495 7194 leader_election.cc:304] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.661787 7357 raft_consensus.cc:2798] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250111 02:54:13.661929 7357 raft_consensus.cc:491] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250111 02:54:13.662142 7357 raft_consensus.cc:3054] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.664886 7357 raft_consensus.cc:513] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.666160 7355 consensus_queue.cc:1035] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:13.667260 7153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ee7290ae1847c390db93c34bef48bf" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:13.667505 7153 raft_consensus.cc:3054] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.669286 7153 raft_consensus.cc:2463] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 1.
I20250111 02:54:13.669961 7194 leader_election.cc:304] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:13.670266 7357 leader_election.cc:290] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 1 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:13.670588 7357 raft_consensus.cc:2798] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader election won for term 1
I20250111 02:54:13.670730 7357 raft_consensus.cc:695] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 1 LEADER]: Becoming Leader. State: Replica: ec1ece0a31324840be298ddc7e853d1b, State: Running, Role: LEADER
I20250111 02:54:13.670851 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0ee7290ae1847c390db93c34bef48bf" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
I20250111 02:54:13.671046 7301 raft_consensus.cc:3054] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:13.671041 7357 consensus_queue.cc:237] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.673378 7301 raft_consensus.cc:2463] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 1.
I20250111 02:54:13.675022 7051 catalog_manager.cc:5526] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b reported cstate change: term changed from 0 to 1, leader changed from <none> to ec1ece0a31324840be298ddc7e853d1b (127.0.103.2). New cstate: current_term: 1 leader_uuid: "ec1ece0a31324840be298ddc7e853d1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:13.679075 412 tablet_server.cc:178] TabletServer@127.0.103.1:0 shutting down...
I20250111 02:54:13.685240 7355 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:13.688944 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:13.689518 412 tablet_replica.cc:331] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: stopping tablet replica
I20250111 02:54:13.689908 412 raft_consensus.cc:2238] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:13.690965 7366 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:13.691642 412 raft_consensus.cc:2267] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:13.692726 412 tablet_replica.cc:331] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: stopping tablet replica
I20250111 02:54:13.692981 412 raft_consensus.cc:2238] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:13.693202 412 raft_consensus.cc:2267] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:13.693948 412 tablet_replica.cc:331] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: stopping tablet replica
I20250111 02:54:13.694223 412 raft_consensus.cc:2238] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:13.694406 412 raft_consensus.cc:2267] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:13.695072 412 tablet_replica.cc:331] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: stopping tablet replica
I20250111 02:54:13.695267 412 raft_consensus.cc:2238] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:13.695448 412 raft_consensus.cc:2267] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:13.696230 412 tablet_replica.cc:331] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: stopping tablet replica
I20250111 02:54:13.696395 412 raft_consensus.cc:2238] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:13.696676 412 raft_consensus.cc:2267] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:13.697373 412 tablet_replica.cc:331] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: stopping tablet replica
I20250111 02:54:13.697526 412 raft_consensus.cc:2238] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:13.697806 412 raft_consensus.cc:2267] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:13.698426 412 tablet_replica.cc:331] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: stopping tablet replica
I20250111 02:54:13.698606 412 raft_consensus.cc:2238] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:13.698824 412 raft_consensus.cc:2267] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:13.699270 412 tablet_replica.cc:331] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: stopping tablet replica
I20250111 02:54:13.699380 412 raft_consensus.cc:2238] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:13.699496 412 raft_consensus.cc:2267] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:13.705143 412 tablet_server.cc:195] TabletServer@127.0.103.1:0 shutdown complete.
W20250111 02:54:13.878400 7268 consensus_peers.cc:487] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b -> Peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): Couldn't send request to peer 502ca883a5d74182a320d1d213c3d0c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:13.905673 7352 raft_consensus.cc:491] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250111 02:54:13.905831 7409 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:13.905892 7352 raft_consensus.cc:513] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:13.906531 7352 leader_election.cc:290] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
W20250111 02:54:13.906618 7268 consensus_peers.cc:487] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b -> Peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): Couldn't send request to peer 502ca883a5d74182a320d1d213c3d0c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:13.907022 7227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5974446b8044cb5b2146db5ceb7c5a3" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b" is_pre_election: true
W20250111 02:54:13.907481 7268 leader_election.cc:336] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111)
I20250111 02:54:13.907624 7266 leader_election.cc:304] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ddfa5c4f3c534b3b96b0269100068a4b; no voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b
I20250111 02:54:13.907846 7352 raft_consensus.cc:2743] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250111 02:54:13.936152 7409 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:14.036948 7301 raft_consensus.cc:3054] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 0 FOLLOWER]: Advancing to term 1
I20250111 02:54:14.039351 7357 consensus_queue.cc:1035] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:54:14.044898 7194 consensus_peers.cc:487] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b -> Peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): Couldn't send request to peer 502ca883a5d74182a320d1d213c3d0c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250111 02:54:14.075951 7194 consensus_peers.cc:487] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b -> Peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): Couldn't send request to peer 502ca883a5d74182a320d1d213c3d0c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:14.089749 7393 consensus_queue.cc:1035] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:54:14.094127 7194 consensus_peers.cc:487] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b -> Peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): Couldn't send request to peer 502ca883a5d74182a320d1d213c3d0c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:14.108745 7392 consensus_queue.cc:1035] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250111 02:54:14.181428 7194 consensus_peers.cc:487] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b -> Peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): Couldn't send request to peer 502ca883a5d74182a320d1d213c3d0c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:14.191310 7393 consensus_queue.cc:1035] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:14.408157 7411 consensus_queue.cc:579] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Leader has been unable to successfully communicate with peer 502ca883a5d74182a320d1d213c3d0c0 for more than 1 seconds (1.114s)
I20250111 02:54:14.472162 7411 consensus_queue.cc:579] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Leader has been unable to successfully communicate with peer 502ca883a5d74182a320d1d213c3d0c0 for more than 1 seconds (1.093s)
I20250111 02:54:14.717820 7050 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35766:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250111 02:54:14.718269 7050 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35766:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250111 02:54:14.723027 7050 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: Sending DeleteTablet for 3 replicas of tablet 10c7b479b15c4335912eb49e40aec3d0
I20250111 02:54:14.723604 7050 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: Sending DeleteTablet for 3 replicas of tablet b0ee7290ae1847c390db93c34bef48bf
I20250111 02:54:14.724009 7216 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c7b479b15c4335912eb49e40aec3d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:56192
I20250111 02:54:14.724124 7050 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: Sending DeleteTablet for 3 replicas of tablet f664c6e13c3a46ea927f0e9713ef6b91
I20250111 02:54:14.724009 7289 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c7b479b15c4335912eb49e40aec3d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:50908
I20250111 02:54:14.724509 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ee7290ae1847c390db93c34bef48bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:50908
I20250111 02:54:14.724656 7050 catalog_manager.cc:5813] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: Sending DeleteTablet for 3 replicas of tablet d5974446b8044cb5b2146db5ceb7c5a3
I20250111 02:54:14.724967 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet f664c6e13c3a46ea927f0e9713ef6b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:50908
I20250111 02:54:14.725121 7218 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ee7290ae1847c390db93c34bef48bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:56192
I20250111 02:54:14.725327 7215 tablet_service.cc:1514] Processing DeleteTablet for tablet f664c6e13c3a46ea927f0e9713ef6b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:56192
I20250111 02:54:14.725921 7428 tablet_replica.cc:331] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: stopping tablet replica
I20250111 02:54:14.725921 7216 tablet_service.cc:1514] Processing DeleteTablet for tablet d5974446b8044cb5b2146db5ceb7c5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:56192
I20250111 02:54:14.726305 7428 raft_consensus.cc:2238] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:14.726619 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250111 02:54:14.726841 7428 raft_consensus.cc:2267] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.728011 7428 ts_tablet_manager.cc:1905] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.728564 7292 tablet_service.cc:1514] Processing DeleteTablet for tablet d5974446b8044cb5b2146db5ceb7c5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:50908
I20250111 02:54:14.735834 7428 ts_tablet_manager.cc:1918] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:54:14.736065 7428 log.cc:1198] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root/wals/10c7b479b15c4335912eb49e40aec3d0
I20250111 02:54:14.736591 7429 tablet_replica.cc:331] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: stopping tablet replica
I20250111 02:54:14.736642 7428 ts_tablet_manager.cc:1939] T 10c7b479b15c4335912eb49e40aec3d0 P ec1ece0a31324840be298ddc7e853d1b: Deleting consensus metadata
W20250111 02:54:14.736696 7430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:14.737632 7038 catalog_manager.cc:4615] TS 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0ee7290ae1847c390db93c34bef48bf: Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111)
W20250111 02:54:14.737955 7432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:14.738119 7429 raft_consensus.cc:2238] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:14.738297 7429 raft_consensus.cc:2267] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.738756 7036 catalog_manager.cc:4872] TS ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955): tablet 10c7b479b15c4335912eb49e40aec3d0 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.739120 7428 tablet_replica.cc:331] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: stopping tablet replica
I20250111 02:54:14.739306 7428 raft_consensus.cc:2238] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 1 LEADER]: Raft consensus shutting down.
W20250111 02:54:14.739364 7435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:14.739313 7429 ts_tablet_manager.cc:1905] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.739709 7428 raft_consensus.cc:2267] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.740343 7428 ts_tablet_manager.cc:1905] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.740551 412 server_base.cc:1034] running on GCE node
I20250111 02:54:14.740900 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:14.740999 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:14.741082 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564054741073 us; error 0 us; skew 500 ppm
I20250111 02:54:14.741284 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:14.742998 412 webserver.cc:458] Webserver started at http://127.0.103.4:39891/ using document root <none> and password file <none>
I20250111 02:54:14.743310 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:14.743422 412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250111 02:54:14.743603 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250111 02:54:14.744349 412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-3-root/instance:
uuid: "bf12e8128df5454699ffa9f896018b01"
format_stamp: "Formatted at 2025-01-11 02:54:14 on dist-test-slave-4xrj"
I20250111 02:54:14.744683 7428 ts_tablet_manager.cc:1918] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:54:14.744776 7428 log.cc:1198] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root/wals/b0ee7290ae1847c390db93c34bef48bf
I20250111 02:54:14.745146 7428 ts_tablet_manager.cc:1939] T b0ee7290ae1847c390db93c34bef48bf P ec1ece0a31324840be298ddc7e853d1b: Deleting consensus metadata
I20250111 02:54:14.745325 7429 ts_tablet_manager.cc:1918] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:54:14.745465 7429 log.cc:1198] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root/wals/10c7b479b15c4335912eb49e40aec3d0
I20250111 02:54:14.745805 7429 ts_tablet_manager.cc:1939] T 10c7b479b15c4335912eb49e40aec3d0 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting consensus metadata
I20250111 02:54:14.746140 7036 catalog_manager.cc:4872] TS ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955): tablet b0ee7290ae1847c390db93c34bef48bf (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.746374 7428 tablet_replica.cc:331] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: stopping tablet replica
I20250111 02:54:14.746560 7428 raft_consensus.cc:2238] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:14.746721 7429 tablet_replica.cc:331] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: stopping tablet replica
I20250111 02:54:14.746788 7037 catalog_manager.cc:4872] TS ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345): tablet 10c7b479b15c4335912eb49e40aec3d0 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.746954 7428 raft_consensus.cc:2267] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.746937 7429 raft_consensus.cc:2238] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:14.747138 7429 raft_consensus.cc:2267] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.747695 7428 ts_tablet_manager.cc:1905] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.748245 7429 ts_tablet_manager.cc:1905] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.751578 7428 ts_tablet_manager.cc:1918] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:54:14.751670 7428 log.cc:1198] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root/wals/f664c6e13c3a46ea927f0e9713ef6b91
I20250111 02:54:14.751974 7428 ts_tablet_manager.cc:1939] T f664c6e13c3a46ea927f0e9713ef6b91 P ec1ece0a31324840be298ddc7e853d1b: Deleting consensus metadata
I20250111 02:54:14.752725 7428 tablet_replica.cc:331] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: stopping tablet replica
I20250111 02:54:14.752852 7036 catalog_manager.cc:4872] TS ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955): tablet f664c6e13c3a46ea927f0e9713ef6b91 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.752875 7428 raft_consensus.cc:2238] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:14.753158 7428 raft_consensus.cc:2267] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.753811 7428 ts_tablet_manager.cc:1905] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.755892 7429 ts_tablet_manager.cc:1918] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:54:14.756039 7429 log.cc:1198] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root/wals/b0ee7290ae1847c390db93c34bef48bf
I20250111 02:54:14.756444 7429 ts_tablet_manager.cc:1939] T b0ee7290ae1847c390db93c34bef48bf P ddfa5c4f3c534b3b96b0269100068a4b: Deleting consensus metadata
I20250111 02:54:14.756528 412 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.004s sys 0.008s
I20250111 02:54:14.757299 7429 tablet_replica.cc:331] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: stopping tablet replica
I20250111 02:54:14.757419 7037 catalog_manager.cc:4872] TS ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345): tablet b0ee7290ae1847c390db93c34bef48bf (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.757529 7429 raft_consensus.cc:2238] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:14.757670 7429 raft_consensus.cc:2267] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.758013 7428 ts_tablet_manager.cc:1918] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:54:14.758179 7428 log.cc:1198] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root/wals/d5974446b8044cb5b2146db5ceb7c5a3
I20250111 02:54:14.758502 7429 ts_tablet_manager.cc:1905] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.758503 7428 ts_tablet_manager.cc:1939] T d5974446b8044cb5b2146db5ceb7c5a3 P ec1ece0a31324840be298ddc7e853d1b: Deleting consensus metadata
I20250111 02:54:14.759272 7036 catalog_manager.cc:4872] TS ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955): tablet d5974446b8044cb5b2146db5ceb7c5a3 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.762142 7439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:14.762516 412 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250111 02:54:14.762600 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-3-root
uuid: "bf12e8128df5454699ffa9f896018b01"
format_stamp: "Formatted at 2025-01-11 02:54:14 on dist-test-slave-4xrj"
I20250111 02:54:14.762698 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:14.764587 7429 ts_tablet_manager.cc:1918] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:54:14.764680 7429 log.cc:1198] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root/wals/f664c6e13c3a46ea927f0e9713ef6b91
I20250111 02:54:14.765002 7429 ts_tablet_manager.cc:1939] T f664c6e13c3a46ea927f0e9713ef6b91 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting consensus metadata
I20250111 02:54:14.765908 7429 tablet_replica.cc:331] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: stopping tablet replica
I20250111 02:54:14.766085 7429 raft_consensus.cc:2238] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:14.766217 7037 catalog_manager.cc:4872] TS ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345): tablet f664c6e13c3a46ea927f0e9713ef6b91 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.766266 7429 raft_consensus.cc:2267] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.766986 7429 ts_tablet_manager.cc:1905] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.770848 7429 ts_tablet_manager.cc:1918] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250111 02:54:14.770941 7429 log.cc:1198] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root/wals/d5974446b8044cb5b2146db5ceb7c5a3
I20250111 02:54:14.771241 7429 ts_tablet_manager.cc:1939] T d5974446b8044cb5b2146db5ceb7c5a3 P ddfa5c4f3c534b3b96b0269100068a4b: Deleting consensus metadata
I20250111 02:54:14.772066 7037 catalog_manager.cc:4872] TS ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345): tablet d5974446b8044cb5b2146db5ceb7c5a3 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.774156 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:14.774601 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:14.775094 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:14.775898 412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250111 02:54:14.775951 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:14.776031 412 ts_tablet_manager.cc:610] Registered 0 tablets
I20250111 02:54:14.776098 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:14.787295 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.4:36829
I20250111 02:54:14.787323 7501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.4:36829 every 8 connection(s)
I20250111 02:54:14.788897 412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250111 02:54:14.791827 7505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250111 02:54:14.792943 7506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:14.795306 7502 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43731
I20250111 02:54:14.795433 7502 heartbeater.cc:463] Registering TS with master...
I20250111 02:54:14.795727 7502 heartbeater.cc:510] Master 127.0.103.62:43731 requested a full tablet report, sending...
I20250111 02:54:14.796779 7050 ts_manager.cc:194] Registered new tserver with Master: bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829)
I20250111 02:54:14.797518 412 server_base.cc:1034] running on GCE node
I20250111 02:54:14.797739 7050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41260
W20250111 02:54:14.798599 7508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250111 02:54:14.798938 412 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250111 02:54:14.798993 412 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250111 02:54:14.799029 412 hybrid_clock.cc:648] HybridClock initialized: now 1736564054799026 us; error 0 us; skew 500 ppm
I20250111 02:54:14.799146 412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250111 02:54:14.800326 412 webserver.cc:458] Webserver started at http://127.0.103.1:44679/ using document root <none> and password file <none>
I20250111 02:54:14.800514 412 fs_manager.cc:362] Metadata directory not provided
I20250111 02:54:14.800554 412 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250111 02:54:14.802644 412 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:14.803017 7301 consensus_queue.cc:237] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:14.804716 7513 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250111 02:54:14.805281 7228 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:54:14.805917 7411 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:54:14.806684 412 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
W20250111 02:54:14.806663 7268 consensus_peers.cc:487] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b -> Peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): Couldn't send request to peer 502ca883a5d74182a320d1d213c3d0c0. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.1:37843: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:14.806835 412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root
uuid: "502ca883a5d74182a320d1d213c3d0c0"
format_stamp: "Formatted at 2025-01-11 02:54:11 on dist-test-slave-4xrj"
I20250111 02:54:14.806944 412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250111 02:54:14.808048 7352 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:14.811144 7037 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250111 02:54:14.811904 7228 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
W20250111 02:54:14.814708 7265 consensus_peers.cc:487] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b -> Peer bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829): Couldn't send request to peer bf12e8128df5454699ffa9f896018b01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35ccdf9e9bdf46179da1c099ea14c3b6. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:14.815275 7050 catalog_manager.cc:5526] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: config changed from index -1 to 2, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:14.825803 412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250111 02:54:14.826318 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250111 02:54:14.826836 412 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250111 02:54:14.827948 7522 ts_tablet_manager.cc:542] Loading tablet metadata (0/8 complete)
I20250111 02:54:14.840844 412 ts_tablet_manager.cc:579] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20250111 02:54:14.840921 412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.000s sys 0.001s
I20250111 02:54:14.840973 412 ts_tablet_manager.cc:594] Registering tablets (0/8 complete)
I20250111 02:54:14.842715 7522 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:14.846923 7522 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:14.847302 7522 tablet_bootstrap.cc:492] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap complete.
I20250111 02:54:14.847543 7522 ts_tablet_manager.cc:1397] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250111 02:54:14.848178 7522 raft_consensus.cc:357] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.848366 7522 raft_consensus.cc:738] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:14.848572 7522 consensus_queue.cc:260] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.849234 7522 ts_tablet_manager.cc:1428] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250111 02:54:14.849469 7522 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:14.853528 7522 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:14.853919 7522 tablet_bootstrap.cc:492] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap complete.
I20250111 02:54:14.854176 7522 ts_tablet_manager.cc:1397] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250111 02:54:14.854903 7522 raft_consensus.cc:357] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.855116 7522 raft_consensus.cc:738] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:14.855331 7522 consensus_queue.cc:260] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.855813 7522 ts_tablet_manager.cc:1428] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250111 02:54:14.856014 7522 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:14.858204 412 ts_tablet_manager.cc:610] Registered 8 tablets
I20250111 02:54:14.858275 412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.017s user 0.017s sys 0.000s
I20250111 02:54:14.861253 7522 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:14.861788 7522 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap complete.
I20250111 02:54:14.862124 7522 ts_tablet_manager.cc:1397] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20250111 02:54:14.863041 7522 raft_consensus.cc:357] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:14.863313 7522 raft_consensus.cc:738] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:14.863572 7522 consensus_queue.cc:260] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:14.864193 7522 ts_tablet_manager.cc:1428] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250111 02:54:14.864519 7522 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:14.869692 7522 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:14.870034 7522 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap complete.
I20250111 02:54:14.870251 7522 ts_tablet_manager.cc:1397] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20250111 02:54:14.870828 7522 raft_consensus.cc:357] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.870986 7522 raft_consensus.cc:738] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:14.871218 7522 consensus_queue.cc:260] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.871649 7522 ts_tablet_manager.cc:1428] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:54:14.871852 7522 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:14.875847 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.1:37843
I20250111 02:54:14.875864 7522 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:14.875864 7584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.1:37843 every 8 connection(s)
I20250111 02:54:14.876296 7522 tablet_bootstrap.cc:492] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap complete.
I20250111 02:54:14.876490 7522 ts_tablet_manager.cc:1397] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250111 02:54:14.877025 7522 raft_consensus.cc:357] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.877213 7522 raft_consensus.cc:738] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:14.877449 7522 consensus_queue.cc:260] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.878075 7522 ts_tablet_manager.cc:1428] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:14.885161 7585 heartbeater.cc:346] Connected to a master server at 127.0.103.62:43731
I20250111 02:54:14.885301 7585 heartbeater.cc:463] Registering TS with master...
W20250111 02:54:14.885565 7099 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 502ca883a5d74182a320d1d213c3d0c0 not available for placement
I20250111 02:54:14.885782 7585 heartbeater.cc:510] Master 127.0.103.62:43731 requested a full tablet report, sending...
I20250111 02:54:14.887106 7051 ts_manager.cc:194] Re-registered known tserver with Master: 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843)
I20250111 02:54:14.889648 7051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41268
I20250111 02:54:14.890776 7585 heartbeater.cc:502] Master 127.0.103.62:43731 was elected leader, sending a full tablet report...
I20250111 02:54:14.891968 7522 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:14.895195 7522 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:14.895548 7522 tablet_bootstrap.cc:492] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap complete.
I20250111 02:54:14.895804 7522 ts_tablet_manager.cc:1397] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:54:14.896351 7522 raft_consensus.cc:357] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.896502 7522 raft_consensus.cc:738] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:14.896723 7522 consensus_queue.cc:260] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.897089 7522 ts_tablet_manager.cc:1428] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:54:14.897279 7522 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:14.900816 7522 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:14.901085 7522 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap complete.
I20250111 02:54:14.901319 7522 ts_tablet_manager.cc:1397] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:54:14.901806 7522 raft_consensus.cc:357] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.901985 7522 raft_consensus.cc:738] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:14.902189 7522 consensus_queue.cc:260] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.902601 7522 ts_tablet_manager.cc:1428] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250111 02:54:14.902810 7522 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap starting.
I20250111 02:54:14.906316 7522 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:14.906687 7522 tablet_bootstrap.cc:492] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Bootstrap complete.
I20250111 02:54:14.906931 7522 ts_tablet_manager.cc:1397] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250111 02:54:14.907505 7522 raft_consensus.cc:357] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.907712 7522 raft_consensus.cc:738] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Initialized, Role: FOLLOWER
I20250111 02:54:14.907915 7522 consensus_queue.cc:260] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:14.908370 7522 ts_tablet_manager.cc:1428] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250111 02:54:14.911617 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet 10c7b479b15c4335912eb49e40aec3d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:54696
I20250111 02:54:14.912107 7592 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:14.912298 7592 raft_consensus.cc:2238] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:14.912438 7592 raft_consensus.cc:2267] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.913012 7592 ts_tablet_manager.cc:1905] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.917079 7592 ts_tablet_manager.cc:1918] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:54:14.917177 7592 log.cc:1198] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root/wals/10c7b479b15c4335912eb49e40aec3d0
I20250111 02:54:14.917466 7592 ts_tablet_manager.cc:1939] T 10c7b479b15c4335912eb49e40aec3d0 P 502ca883a5d74182a320d1d213c3d0c0: Deleting consensus metadata
I20250111 02:54:14.918366 7038 catalog_manager.cc:4872] TS 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): tablet 10c7b479b15c4335912eb49e40aec3d0 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.921092 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet d5974446b8044cb5b2146db5ceb7c5a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:54696
I20250111 02:54:14.921324 7592 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:14.921483 7592 raft_consensus.cc:2238] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:14.921624 7592 raft_consensus.cc:2267] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.922235 7592 ts_tablet_manager.cc:1905] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.925990 7592 ts_tablet_manager.cc:1918] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:54:14.926072 7592 log.cc:1198] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root/wals/d5974446b8044cb5b2146db5ceb7c5a3
I20250111 02:54:14.926333 7592 ts_tablet_manager.cc:1939] T d5974446b8044cb5b2146db5ceb7c5a3 P 502ca883a5d74182a320d1d213c3d0c0: Deleting consensus metadata
I20250111 02:54:14.927107 7038 catalog_manager.cc:4872] TS 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): tablet d5974446b8044cb5b2146db5ceb7c5a3 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.929911 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet f664c6e13c3a46ea927f0e9713ef6b91 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:54696
I20250111 02:54:14.930163 7592 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:14.930325 7592 raft_consensus.cc:2238] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:14.930459 7592 raft_consensus.cc:2267] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.931072 7592 ts_tablet_manager.cc:1905] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.934752 7592 ts_tablet_manager.cc:1918] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:54:14.934840 7592 log.cc:1198] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root/wals/f664c6e13c3a46ea927f0e9713ef6b91
I20250111 02:54:14.935060 7592 ts_tablet_manager.cc:1939] T f664c6e13c3a46ea927f0e9713ef6b91 P 502ca883a5d74182a320d1d213c3d0c0: Deleting consensus metadata
I20250111 02:54:14.936041 7038 catalog_manager.cc:4872] TS 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): tablet f664c6e13c3a46ea927f0e9713ef6b91 (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:14.963220 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ee7290ae1847c390db93c34bef48bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-11 02:54:14 UTC) from {username='slave'} at 127.0.0.1:54696
I20250111 02:54:14.963593 7592 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:14.963775 7592 raft_consensus.cc:2238] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:14.963966 7592 raft_consensus.cc:2267] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:14.964644 7592 ts_tablet_manager.cc:1905] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250111 02:54:14.969195 7592 ts_tablet_manager.cc:1918] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250111 02:54:14.969313 7592 log.cc:1198] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root/wals/b0ee7290ae1847c390db93c34bef48bf
I20250111 02:54:14.969552 7592 ts_tablet_manager.cc:1939] T b0ee7290ae1847c390db93c34bef48bf P 502ca883a5d74182a320d1d213c3d0c0: Deleting consensus metadata
I20250111 02:54:14.970634 7038 catalog_manager.cc:4872] TS 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): tablet b0ee7290ae1847c390db93c34bef48bf (table dugtrio [id=5c3698e179354ce18ac293a1c9f760b2]) successfully deleted
I20250111 02:54:15.077646 7301 consensus_queue.cc:237] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:15.080119 7228 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250111 02:54:15.080242 7265 consensus_peers.cc:487] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b -> Peer bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829): Couldn't send request to peer bf12e8128df5454699ffa9f896018b01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0c1d0c792c4443c962a2a013e49e3af. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:15.080669 7411 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:54:15.083034 7409 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.083679 7228 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.085129 7558 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250111 02:54:15.085740 7411 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:15.086874 7037 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a0c1d0c792c4443c962a2a013e49e3af with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250111 02:54:15.088163 7051 catalog_manager.cc:5526] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b reported cstate change: config changed from index -1 to 2, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.088377 7558 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.166134 7352 raft_consensus.cc:491] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 502ca883a5d74182a320d1d213c3d0c0)
I20250111 02:54:15.166344 7352 raft_consensus.cc:513] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:15.166991 7352 leader_election.cc:290] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:15.167506 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b" is_pre_election: true
I20250111 02:54:15.167507 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:15.167702 7228 raft_consensus.cc:2463] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 1.
I20250111 02:54:15.167737 7558 raft_consensus.cc:2463] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 1.
I20250111 02:54:15.168110 7266 leader_election.cc:304] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:15.168368 7352 raft_consensus.cc:2798] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:54:15.168450 7352 raft_consensus.cc:491] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 502ca883a5d74182a320d1d213c3d0c0)
I20250111 02:54:15.168526 7352 raft_consensus.cc:3054] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:15.170521 7352 raft_consensus.cc:513] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:15.171118 7352 leader_election.cc:290] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 2 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:15.171527 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b"
I20250111 02:54:15.171528 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85776db200cd43268074d11d9c659f79" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:15.171691 7228 raft_consensus.cc:3054] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:15.171715 7558 raft_consensus.cc:3054] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:15.173480 7228 raft_consensus.cc:2463] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 2.
I20250111 02:54:15.173554 7558 raft_consensus.cc:2463] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 2.
I20250111 02:54:15.173825 7266 leader_election.cc:304] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddfa5c4f3c534b3b96b0269100068a4b, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:15.174049 7352 raft_consensus.cc:2798] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:54:15.174176 7352 raft_consensus.cc:695] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Becoming Leader. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Running, Role: LEADER
I20250111 02:54:15.174391 7352 consensus_queue.cc:237] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:15.176725 7050 catalog_manager.cc:5526] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: term changed from 1 to 2, leader changed from 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1) to ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3). New cstate: current_term: 2 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } }
I20250111 02:54:15.191421 7352 raft_consensus.cc:491] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 502ca883a5d74182a320d1d213c3d0c0)
I20250111 02:54:15.191601 7352 raft_consensus.cc:513] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:15.192222 7352 leader_election.cc:290] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:15.192224 7393 raft_consensus.cc:491] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 502ca883a5d74182a320d1d213c3d0c0)
I20250111 02:54:15.192474 7393 raft_consensus.cc:513] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:15.192704 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:15.192728 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b" is_pre_election: true
I20250111 02:54:15.192905 7558 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 1.
I20250111 02:54:15.192932 7228 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 1.
I20250111 02:54:15.193138 7393 leader_election.cc:290] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:15.193372 7268 leader_election.cc:304] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b; no voters:
I20250111 02:54:15.193501 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" is_pre_election: true
I20250111 02:54:15.193665 7301 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec1ece0a31324840be298ddc7e853d1b in term 1.
I20250111 02:54:15.193778 7352 raft_consensus.cc:2798] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:54:15.193885 7352 raft_consensus.cc:491] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 502ca883a5d74182a320d1d213c3d0c0)
I20250111 02:54:15.193969 7352 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:15.194082 7193 leader_election.cc:304] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ddfa5c4f3c534b3b96b0269100068a4b, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:15.194319 7393 raft_consensus.cc:2798] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250111 02:54:15.194399 7393 raft_consensus.cc:491] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 502ca883a5d74182a320d1d213c3d0c0)
I20250111 02:54:15.194470 7393 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:15.196324 7393 raft_consensus.cc:513] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:15.196343 7352 raft_consensus.cc:513] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:15.196915 7393 leader_election.cc:290] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 2 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:15.197173 7352 leader_election.cc:290] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 2 election: Requested vote from peers 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:15.197335 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:15.197433 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
I20250111 02:54:15.197539 7558 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Advancing to term 2
I20250111 02:54:15.197638 7301 raft_consensus.cc:2388] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ec1ece0a31324840be298ddc7e853d1b in current term 2: Already voted for candidate ddfa5c4f3c534b3b96b0269100068a4b in this term.
I20250111 02:54:15.197644 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ec1ece0a31324840be298ddc7e853d1b"
I20250111 02:54:15.197883 7228 raft_consensus.cc:2388] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ddfa5c4f3c534b3b96b0269100068a4b in current term 2: Already voted for candidate ec1ece0a31324840be298ddc7e853d1b in this term.
I20250111 02:54:15.199492 7557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0" is_pre_election: true
I20250111 02:54:15.199568 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "ec1ece0a31324840be298ddc7e853d1b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
I20250111 02:54:15.199990 7558 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddfa5c4f3c534b3b96b0269100068a4b in term 2.
I20250111 02:54:15.200165 7194 leader_election.cc:304] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec1ece0a31324840be298ddc7e853d1b; no voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b
I20250111 02:54:15.200281 7268 leader_election.cc:304] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ddfa5c4f3c534b3b96b0269100068a4b; no voters: ec1ece0a31324840be298ddc7e853d1b
I20250111 02:54:15.200474 7393 raft_consensus.cc:2743] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250111 02:54:15.200475 7352 raft_consensus.cc:2798] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 FOLLOWER]: Leader election won for term 2
I20250111 02:54:15.200692 7352 raft_consensus.cc:695] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Becoming Leader. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Running, Role: LEADER
I20250111 02:54:15.200945 7352 consensus_queue.cc:237] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } }
I20250111 02:54:15.203497 7050 catalog_manager.cc:5526] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: term changed from 1 to 2, leader changed from 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1) to ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3). New cstate: current_term: 2 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:15.418291 7600 ts_tablet_manager.cc:927] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Initiating tablet copy from peer ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:15.418823 7600 tablet_copy_client.cc:323] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:37345
I20250111 02:54:15.424302 7312 tablet_copy_service.cc:140] P ddfa5c4f3c534b3b96b0269100068a4b: Received BeginTabletCopySession request for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 from peer bf12e8128df5454699ffa9f896018b01 ({username='slave'} at 127.0.0.1:45342)
I20250111 02:54:15.424302 7352 consensus_queue.cc:1037] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250111 02:54:15.424503 7312 tablet_copy_service.cc:161] P ddfa5c4f3c534b3b96b0269100068a4b: Beginning new tablet copy session on tablet 35ccdf9e9bdf46179da1c099ea14c3b6 from peer bf12e8128df5454699ffa9f896018b01 at {username='slave'} at 127.0.0.1:45342: session id = bf12e8128df5454699ffa9f896018b01-35ccdf9e9bdf46179da1c099ea14c3b6
I20250111 02:54:15.426801 7312 tablet_copy_source_session.cc:215] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:54:15.426692 7558 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.427906 7600 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ccdf9e9bdf46179da1c099ea14c3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:15.433537 7600 tablet_copy_client.cc:806] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet copy: Starting download of 0 data blocks...
I20250111 02:54:15.433772 7600 tablet_copy_client.cc:670] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet copy: Starting download of 1 WAL segments...
I20250111 02:54:15.435035 7600 tablet_copy_client.cc:538] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:54:15.437216 7600 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Bootstrap starting.
I20250111 02:54:15.441506 7600 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:15.441778 7600 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Bootstrap complete.
I20250111 02:54:15.442001 7600 ts_tablet_manager.cc:1397] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Time spent bootstrapping tablet: real 0.005s user 0.007s sys 0.000s
I20250111 02:54:15.442534 7600 raft_consensus.cc:357] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:15.442703 7600 raft_consensus.cc:738] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf12e8128df5454699ffa9f896018b01, State: Initialized, Role: LEARNER
I20250111 02:54:15.442832 7600 consensus_queue.cc:260] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:15.443411 7600 ts_tablet_manager.cc:1428] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250111 02:54:15.443493 7502 heartbeater.cc:502] Master 127.0.103.62:43731 was elected leader, sending a full tablet report...
I20250111 02:54:15.444113 7312 tablet_copy_service.cc:342] P ddfa5c4f3c534b3b96b0269100068a4b: Request end of tablet copy session bf12e8128df5454699ffa9f896018b01-35ccdf9e9bdf46179da1c099ea14c3b6 received from {username='slave'} at 127.0.0.1:45342
I20250111 02:54:15.444218 7312 tablet_copy_service.cc:434] P ddfa5c4f3c534b3b96b0269100068a4b: ending tablet copy session bf12e8128df5454699ffa9f896018b01-35ccdf9e9bdf46179da1c099ea14c3b6 on tablet 35ccdf9e9bdf46179da1c099ea14c3b6 with peer bf12e8128df5454699ffa9f896018b01
I20250111 02:54:15.518739 7301 consensus_queue.cc:237] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:15.520349 7558 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:54:15.520362 7228 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:54:15.520854 7352 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:15.521045 7411 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:15.523131 7409 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } }
I20250111 02:54:15.523818 7558 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } }
I20250111 02:54:15.526800 7228 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } }
I20250111 02:54:15.528214 7050 catalog_manager.cc:5526] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 reported cstate change: config changed from index 2 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) evicted. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } }
I20250111 02:54:15.529933 7037 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a0c1d0c792c4443c962a2a013e49e3af with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:54:15.542501 7467 tablet_service.cc:1514] Processing DeleteTablet for tablet a0c1d0c792c4443c962a2a013e49e3af with delete_type TABLET_DATA_TOMBSTONED (TS bf12e8128df5454699ffa9f896018b01 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46886
W20250111 02:54:15.543444 7035 catalog_manager.cc:4839] TS bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829): delete failed for tablet a0c1d0c792c4443c962a2a013e49e3af because the tablet was not found. No further retry: Not found: Tablet not found: a0c1d0c792c4443c962a2a013e49e3af
I20250111 02:54:15.591817 7228 raft_consensus.cc:1270] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:54:15.592352 7409 consensus_queue.cc:1035] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:54:15.595357 7558 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:54:15.595928 7409 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:54:15.596057 7558 raft_consensus.cc:1270] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:54:15.597774 7409 consensus_queue.cc:1035] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:54:15.600253 7228 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250111 02:54:15.600907 7411 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250111 02:54:15.875909 7301 consensus_queue.cc:237] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } }
I20250111 02:54:15.877849 7558 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:54:15.877931 7228 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250111 02:54:15.878317 7352 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:15.878538 7411 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:15.880251 7352 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } }
I20250111 02:54:15.881088 7228 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } }
I20250111 02:54:15.883050 7037 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:54:15.883306 7558 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } }
I20250111 02:54:15.884994 7050 catalog_manager.cc:5526] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b reported cstate change: config changed from index 2 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) evicted. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } }
I20250111 02:54:15.891577 7467 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 with delete_type TABLET_DATA_TOMBSTONED (TS bf12e8128df5454699ffa9f896018b01 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:46886
I20250111 02:54:15.891872 7609 tablet_replica.cc:331] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: stopping tablet replica
I20250111 02:54:15.892042 7609 raft_consensus.cc:2238] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Raft consensus shutting down.
I20250111 02:54:15.892179 7609 raft_consensus.cc:2267] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Raft consensus is shut down!
I20250111 02:54:15.893919 7609 ts_tablet_manager.cc:1905] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:54:15.894891 7301 consensus_queue.cc:237] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:15.896740 7558 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:54:15.896991 7228 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250111 02:54:15.897466 7352 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250111 02:54:15.897657 7409 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250111 02:54:15.897917 7265 consensus_peers.cc:487] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b -> Peer bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829): Couldn't send request to peer bf12e8128df5454699ffa9f896018b01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd66331e08224e0a9b7b0db16d52e8aa. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:15.899472 7609 ts_tablet_manager.cc:1918] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250111 02:54:15.899574 7609 log.cc:1198] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-3-root/wals/35ccdf9e9bdf46179da1c099ea14c3b6
I20250111 02:54:15.899569 7409 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.900194 7035 catalog_manager.cc:4872] TS bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829): tablet 35ccdf9e9bdf46179da1c099ea14c3b6 (table test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]) successfully deleted
I20250111 02:54:15.903061 7049 catalog_manager.cc:5526] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: config changed from index -1 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New cstate: current_term: 2 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:15.903544 7228 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.903034 7558 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.903939 7301 consensus_queue.cc:237] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:15.906483 7227 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250111 02:54:15.906526 7265 consensus_peers.cc:487] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b -> Peer bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829): Couldn't send request to peer bf12e8128df5454699ffa9f896018b01. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:15.906903 7556 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:54:15.908104 7411 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:54:15.908658 7411 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:54:15.911010 7409 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.911546 7228 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.911566 7556 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.915509 7050 catalog_manager.cc:5526] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: config changed from index 3 to 4, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:15.916199 7301 consensus_queue.cc:237] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:15.918224 7228 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250111 02:54:15.918624 7265 consensus_peers.cc:487] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b -> Peer bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829): Couldn't send request to peer bf12e8128df5454699ffa9f896018b01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a0c1d0c792c4443c962a2a013e49e3af. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:15.919255 7409 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:54:15.920099 7556 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250111 02:54:15.920609 7409 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:54:15.921247 7352 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.921772 7228 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.925173 7049 catalog_manager.cc:5526] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b reported cstate change: config changed from index 3 to 4, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.927440 7301 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa"
dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50908
I20250111 02:54:15.927588 7301 raft_consensus.cc:604] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Received request to transfer leadership
I20250111 02:54:15.927620 7556 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER bf12e8128df5454699ffa9f896018b01 (127.0.103.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } } }
I20250111 02:54:15.933351 7301 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa"
dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50908
I20250111 02:54:15.933521 7301 raft_consensus.cc:604] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Received request to transfer leadership
I20250111 02:54:15.934649 7301 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa"
dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50908
I20250111 02:54:15.934772 7301 raft_consensus.cc:604] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Received request to transfer leadership
W20250111 02:54:15.935169 7099 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for bd66331e08224e0a9b7b0db16d52e8aa already in progress
W20250111 02:54:15.935223 7099 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for bd66331e08224e0a9b7b0db16d52e8aa already in progress
I20250111 02:54:16.287173 7620 raft_consensus.cc:988] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: : Instructing follower 502ca883a5d74182a320d1d213c3d0c0 to start an election
I20250111 02:54:16.287362 7411 raft_consensus.cc:1076] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Signalling peer 502ca883a5d74182a320d1d213c3d0c0 to start an election
I20250111 02:54:16.287935 7556 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa"
dest_uuid: "502ca883a5d74182a320d1d213c3d0c0"
from {username='slave'} at 127.0.0.1:54706
I20250111 02:54:16.288123 7556 raft_consensus.cc:491] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250111 02:54:16.288220 7556 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:16.290117 7556 raft_consensus.cc:513] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:16.290956 7556 leader_election.cc:290] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 3 election: Requested vote from peers ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345), ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955)
I20250111 02:54:16.297101 7301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "ddfa5c4f3c534b3b96b0269100068a4b"
I20250111 02:54:16.297431 7228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd66331e08224e0a9b7b0db16d52e8aa" candidate_uuid: "502ca883a5d74182a320d1d213c3d0c0" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "ec1ece0a31324840be298ddc7e853d1b"
I20250111 02:54:16.297573 7228 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:16.297672 7301 raft_consensus.cc:3049] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Stepping down as leader of term 2
I20250111 02:54:16.297750 7301 raft_consensus.cc:738] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ddfa5c4f3c534b3b96b0269100068a4b, State: Running, Role: LEADER
I20250111 02:54:16.298018 7301 consensus_queue.cc:260] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:16.298307 7301 raft_consensus.cc:3054] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 2 FOLLOWER]: Advancing to term 3
I20250111 02:54:16.299292 7228 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 502ca883a5d74182a320d1d213c3d0c0 in term 3.
I20250111 02:54:16.299638 7517 leader_election.cc:304] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 502ca883a5d74182a320d1d213c3d0c0, ec1ece0a31324840be298ddc7e853d1b; no voters:
I20250111 02:54:16.299880 7524 raft_consensus.cc:2798] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 3 FOLLOWER]: Leader election won for term 3
I20250111 02:54:16.300841 7301 raft_consensus.cc:2463] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 502ca883a5d74182a320d1d213c3d0c0 in term 3.
I20250111 02:54:16.301262 7524 raft_consensus.cc:695] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 3 LEADER]: Becoming Leader. State: Replica: 502ca883a5d74182a320d1d213c3d0c0, State: Running, Role: LEADER
I20250111 02:54:16.301509 7524 consensus_queue.cc:237] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:16.303808 7051 catalog_manager.cc:5526] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 reported cstate change: term changed from 2 to 3, leader changed from ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3) to 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1). New cstate: current_term: 3 leader_uuid: "502ca883a5d74182a320d1d213c3d0c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250111 02:54:16.317636 7626 ts_tablet_manager.cc:927] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Initiating tablet copy from peer ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:16.318363 7626 tablet_copy_client.cc:323] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:37345
I20250111 02:54:16.318934 7312 tablet_copy_service.cc:140] P ddfa5c4f3c534b3b96b0269100068a4b: Received BeginTabletCopySession request for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 from peer bf12e8128df5454699ffa9f896018b01 ({username='slave'} at 127.0.0.1:45342)
I20250111 02:54:16.319080 7312 tablet_copy_service.cc:161] P ddfa5c4f3c534b3b96b0269100068a4b: Beginning new tablet copy session on tablet 35ccdf9e9bdf46179da1c099ea14c3b6 from peer bf12e8128df5454699ffa9f896018b01 at {username='slave'} at 127.0.0.1:45342: session id = bf12e8128df5454699ffa9f896018b01-35ccdf9e9bdf46179da1c099ea14c3b6
I20250111 02:54:16.321519 7312 tablet_copy_source_session.cc:215] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:54:16.322398 7626 ts_tablet_manager.cc:1905] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250111 02:54:16.328045 7626 ts_tablet_manager.cc:1918] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20250111 02:54:16.328313 7626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ccdf9e9bdf46179da1c099ea14c3b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:16.330682 7626 tablet_copy_client.cc:806] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet copy: Starting download of 0 data blocks...
I20250111 02:54:16.330987 7626 tablet_copy_client.cc:670] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet copy: Starting download of 1 WAL segments...
I20250111 02:54:16.332273 7626 tablet_copy_client.cc:538] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:54:16.335036 7626 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Bootstrap starting.
I20250111 02:54:16.341831 7626 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:16.342195 7626 tablet_bootstrap.cc:492] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Bootstrap complete.
I20250111 02:54:16.342459 7626 ts_tablet_manager.cc:1397] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Time spent bootstrapping tablet: real 0.007s user 0.008s sys 0.000s
I20250111 02:54:16.343158 7626 raft_consensus.cc:357] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:16.343405 7626 raft_consensus.cc:738] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf12e8128df5454699ffa9f896018b01, State: Initialized, Role: LEARNER
I20250111 02:54:16.343608 7626 consensus_queue.cc:260] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:16.344511 7626 ts_tablet_manager.cc:1428] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:16.345088 7312 tablet_copy_service.cc:342] P ddfa5c4f3c534b3b96b0269100068a4b: Request end of tablet copy session bf12e8128df5454699ffa9f896018b01-35ccdf9e9bdf46179da1c099ea14c3b6 received from {username='slave'} at 127.0.0.1:45342
I20250111 02:54:16.345192 7312 tablet_copy_service.cc:434] P ddfa5c4f3c534b3b96b0269100068a4b: ending tablet copy session bf12e8128df5454699ffa9f896018b01-35ccdf9e9bdf46179da1c099ea14c3b6 on tablet 35ccdf9e9bdf46179da1c099ea14c3b6 with peer bf12e8128df5454699ffa9f896018b01
I20250111 02:54:16.357501 7626 ts_tablet_manager.cc:927] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: Initiating tablet copy from peer ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345)
I20250111 02:54:16.358127 7626 tablet_copy_client.cc:323] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.3:37345
I20250111 02:54:16.358666 7312 tablet_copy_service.cc:140] P ddfa5c4f3c534b3b96b0269100068a4b: Received BeginTabletCopySession request for tablet a0c1d0c792c4443c962a2a013e49e3af from peer bf12e8128df5454699ffa9f896018b01 ({username='slave'} at 127.0.0.1:45342)
I20250111 02:54:16.358769 7312 tablet_copy_service.cc:161] P ddfa5c4f3c534b3b96b0269100068a4b: Beginning new tablet copy session on tablet a0c1d0c792c4443c962a2a013e49e3af from peer bf12e8128df5454699ffa9f896018b01 at {username='slave'} at 127.0.0.1:45342: session id = bf12e8128df5454699ffa9f896018b01-a0c1d0c792c4443c962a2a013e49e3af
I20250111 02:54:16.360358 7312 tablet_copy_source_session.cc:215] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:54:16.361301 7626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0c1d0c792c4443c962a2a013e49e3af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:16.367332 7626 tablet_copy_client.cc:806] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: tablet copy: Starting download of 0 data blocks...
I20250111 02:54:16.367570 7626 tablet_copy_client.cc:670] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: tablet copy: Starting download of 1 WAL segments...
I20250111 02:54:16.370879 7626 tablet_copy_client.cc:538] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:54:16.382732 7626 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: Bootstrap starting.
I20250111 02:54:16.388146 7626 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:16.388437 7626 tablet_bootstrap.cc:492] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: Bootstrap complete.
I20250111 02:54:16.388653 7626 ts_tablet_manager.cc:1397] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250111 02:54:16.389154 7626 raft_consensus.cc:357] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:16.389326 7626 raft_consensus.cc:738] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf12e8128df5454699ffa9f896018b01, State: Initialized, Role: LEARNER
I20250111 02:54:16.389559 7626 consensus_queue.cc:260] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:16.390009 7626 ts_tablet_manager.cc:1428] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20250111 02:54:16.390585 7312 tablet_copy_service.cc:342] P ddfa5c4f3c534b3b96b0269100068a4b: Request end of tablet copy session bf12e8128df5454699ffa9f896018b01-a0c1d0c792c4443c962a2a013e49e3af received from {username='slave'} at 127.0.0.1:45342
I20250111 02:54:16.390694 7312 tablet_copy_service.cc:434] P ddfa5c4f3c534b3b96b0269100068a4b: ending tablet copy session bf12e8128df5454699ffa9f896018b01-a0c1d0c792c4443c962a2a013e49e3af on tablet a0c1d0c792c4443c962a2a013e49e3af with peer bf12e8128df5454699ffa9f896018b01
W20250111 02:54:16.709092 7516 consensus_peers.cc:487] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 -> Peer bf12e8128df5454699ffa9f896018b01 (127.0.103.4:36829): Couldn't send request to peer bf12e8128df5454699ffa9f896018b01. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd66331e08224e0a9b7b0db16d52e8aa. This is attempt 1: this message will repeat every 5th retry.
I20250111 02:54:16.811424 7477 raft_consensus.cc:1212] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250111 02:54:16.837945 7477 raft_consensus.cc:1212] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250111 02:54:16.883353 7228 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 3 FOLLOWER]: Refusing update from remote peer 502ca883a5d74182a320d1d213c3d0c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250111 02:54:16.883839 7524 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:54:16.888140 7301 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 3 FOLLOWER]: Refusing update from remote peer 502ca883a5d74182a320d1d213c3d0c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20250111 02:54:16.888720 7524 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250111 02:54:17.138099 7641 ts_tablet_manager.cc:927] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: Initiating tablet copy from peer 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843)
I20250111 02:54:17.138718 7641 tablet_copy_client.cc:323] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.1:37843
I20250111 02:54:17.145258 7569 tablet_copy_service.cc:140] P 502ca883a5d74182a320d1d213c3d0c0: Received BeginTabletCopySession request for tablet bd66331e08224e0a9b7b0db16d52e8aa from peer bf12e8128df5454699ffa9f896018b01 ({username='slave'} at 127.0.0.1:54726)
I20250111 02:54:17.145408 7569 tablet_copy_service.cc:161] P 502ca883a5d74182a320d1d213c3d0c0: Beginning new tablet copy session on tablet bd66331e08224e0a9b7b0db16d52e8aa from peer bf12e8128df5454699ffa9f896018b01 at {username='slave'} at 127.0.0.1:54726: session id = bf12e8128df5454699ffa9f896018b01-bd66331e08224e0a9b7b0db16d52e8aa
I20250111 02:54:17.147730 7569 tablet_copy_source_session.cc:215] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250111 02:54:17.148806 7641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd66331e08224e0a9b7b0db16d52e8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250111 02:54:17.154548 7641 tablet_copy_client.cc:806] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: tablet copy: Starting download of 0 data blocks...
I20250111 02:54:17.154773 7641 tablet_copy_client.cc:670] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: tablet copy: Starting download of 1 WAL segments...
I20250111 02:54:17.156129 7641 tablet_copy_client.cc:538] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250111 02:54:17.159703 7641 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: Bootstrap starting.
I20250111 02:54:17.166716 7641 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250111 02:54:17.167121 7641 tablet_bootstrap.cc:492] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: Bootstrap complete.
I20250111 02:54:17.167385 7641 ts_tablet_manager.cc:1397] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20250111 02:54:17.168046 7641 raft_consensus.cc:357] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:17.168280 7641 raft_consensus.cc:738] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: bf12e8128df5454699ffa9f896018b01, State: Initialized, Role: LEARNER
I20250111 02:54:17.168509 7641 consensus_queue.cc:260] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: NON_VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: true } }
I20250111 02:54:17.169027 7641 ts_tablet_manager.cc:1428] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250111 02:54:17.169945 7569 tablet_copy_service.cc:342] P 502ca883a5d74182a320d1d213c3d0c0: Request end of tablet copy session bf12e8128df5454699ffa9f896018b01-bd66331e08224e0a9b7b0db16d52e8aa received from {username='slave'} at 127.0.0.1:54726
I20250111 02:54:17.170046 7569 tablet_copy_service.cc:434] P 502ca883a5d74182a320d1d213c3d0c0: ending tablet copy session bf12e8128df5454699ffa9f896018b01-bd66331e08224e0a9b7b0db16d52e8aa on tablet bd66331e08224e0a9b7b0db16d52e8aa with peer bf12e8128df5454699ffa9f896018b01
I20250111 02:54:17.220153 7633 raft_consensus.cc:1059] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: attempting to promote NON_VOTER bf12e8128df5454699ffa9f896018b01 to VOTER
I20250111 02:54:17.220861 7633 consensus_queue.cc:237] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } }
I20250111 02:54:17.223258 7477 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250111 02:54:17.223536 7228 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250111 02:54:17.224053 7633 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.224360 7647 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.227596 7633 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.228410 7228 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.230695 7556 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250111 02:54:17.231122 7477 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.231971 7648 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.232867 7051 catalog_manager.cc:5526] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:54:17.237059 7556 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } attrs { replace: true } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.313004 7411 raft_consensus.cc:1059] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b: attempting to promote NON_VOTER bf12e8128df5454699ffa9f896018b01 to VOTER
I20250111 02:54:17.313688 7411 consensus_queue.cc:237] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } }
I20250111 02:54:17.315693 7477 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 LEARNER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250111 02:54:17.315730 7556 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250111 02:54:17.316124 7228 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250111 02:54:17.316205 7648 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.316586 7633 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.316825 7411 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.328366 7633 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.328994 7477 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.331718 7228 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.332450 7050 catalog_manager.cc:5526] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:54:17.332867 7556 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.337428 7301 consensus_queue.cc:237] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } }
I20250111 02:54:17.339398 7477 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250111 02:54:17.339495 7228 raft_consensus.cc:1270] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250111 02:54:17.339864 7647 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250111 02:54:17.340111 7633 consensus_queue.cc:1035] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250111 02:54:17.342171 7648 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.342737 7477 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.342950 7228 raft_consensus.cc:2949] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.345800 7037 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet a0c1d0c792c4443c962a2a013e49e3af with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:54:17.346825 7049 catalog_manager.cc:5526] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 reported cstate change: config changed from index 5 to 6, VOTER 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1) evicted. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.350433 7548 tablet_service.cc:1514] Processing DeleteTablet for tablet a0c1d0c792c4443c962a2a013e49e3af with delete_type TABLET_DATA_TOMBSTONED (TS 502ca883a5d74182a320d1d213c3d0c0 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54696
I20250111 02:54:17.351073 7659 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:17.351308 7659 raft_consensus.cc:2238] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:17.351799 7659 raft_consensus.cc:2267] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:17.352604 7659 ts_tablet_manager.cc:1905] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:54:17.357322 7659 ts_tablet_manager.cc:1918] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250111 02:54:17.357415 7659 log.cc:1198] T a0c1d0c792c4443c962a2a013e49e3af P 502ca883a5d74182a320d1d213c3d0c0: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-0-root/wals/a0c1d0c792c4443c962a2a013e49e3af
I20250111 02:54:17.358040 7038 catalog_manager.cc:4872] TS 502ca883a5d74182a320d1d213c3d0c0 (127.0.103.1:37843): tablet a0c1d0c792c4443c962a2a013e49e3af (table test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]) successfully deleted
I20250111 02:54:17.519177 7477 raft_consensus.cc:1212] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 LEARNER]: Deduplicated request from leader. Original: 2.3->[3.4-3.4] Dedup: 3.4->[]
I20250111 02:54:17.653636 7301 consensus_queue.cc:237] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } }
I20250111 02:54:17.655628 7477 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250111 02:54:17.656123 7634 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250111 02:54:17.656201 7556 raft_consensus.cc:1270] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Refusing update from remote peer ddfa5c4f3c534b3b96b0269100068a4b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250111 02:54:17.656754 7634 consensus_queue.cc:1035] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [LEADER]: Connected to new peer: Peer: permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250111 02:54:17.658394 7648 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec1ece0a31324840be298ddc7e853d1b (127.0.103.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.658949 7477 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec1ece0a31324840be298ddc7e853d1b (127.0.103.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.661222 7037 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:54:17.662549 7556 raft_consensus.cc:2949] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec1ece0a31324840be298ddc7e853d1b (127.0.103.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.663233 7051 catalog_manager.cc:5526] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b reported cstate change: config changed from index 5 to 6, VOTER ec1ece0a31324840be298ddc7e853d1b (127.0.103.2) evicted. New cstate: current_term: 1 leader_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:54:17.667291 7216 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ccdf9e9bdf46179da1c099ea14c3b6 with delete_type TABLET_DATA_TOMBSTONED (TS ec1ece0a31324840be298ddc7e853d1b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56192
I20250111 02:54:17.667949 7662 tablet_replica.cc:331] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: stopping tablet replica
I20250111 02:54:17.668121 7662 raft_consensus.cc:2238] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:17.668332 7662 raft_consensus.cc:2267] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:17.668975 7662 ts_tablet_manager.cc:1905] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:54:17.673064 7662 ts_tablet_manager.cc:1918] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250111 02:54:17.673169 7662 log.cc:1198] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ec1ece0a31324840be298ddc7e853d1b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-1-root/wals/35ccdf9e9bdf46179da1c099ea14c3b6
I20250111 02:54:17.673796 7036 catalog_manager.cc:4872] TS ec1ece0a31324840be298ddc7e853d1b (127.0.103.2:38955): tablet 35ccdf9e9bdf46179da1c099ea14c3b6 (table test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]) successfully deleted
I20250111 02:54:17.900501 7660 raft_consensus.cc:1059] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0: attempting to promote NON_VOTER bf12e8128df5454699ffa9f896018b01 to VOTER
I20250111 02:54:17.901108 7660 consensus_queue.cc:237] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } }
I20250111 02:54:17.903169 7228 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 3 FOLLOWER]: Refusing update from remote peer 502ca883a5d74182a320d1d213c3d0c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250111 02:54:17.903249 7302 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 3 FOLLOWER]: Refusing update from remote peer 502ca883a5d74182a320d1d213c3d0c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250111 02:54:17.903249 7477 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 LEARNER]: Refusing update from remote peer 502ca883a5d74182a320d1d213c3d0c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250111 02:54:17.903748 7664 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.904035 7660 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.904292 7661 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250111 02:54:17.906904 7660 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.907619 7302 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.907832 7477 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.911444 7228 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.911877 7049 catalog_manager.cc:5526] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 reported cstate change: config changed from index 3 to 5, bf12e8128df5454699ffa9f896018b01 (127.0.103.4) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "502ca883a5d74182a320d1d213c3d0c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddfa5c4f3c534b3b96b0269100068a4b" member_type: VOTER last_known_addr { host: "127.0.103.3" port: 37345 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:54:17.919076 7556 consensus_queue.cc:237] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } }
I20250111 02:54:17.920565 7477 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 FOLLOWER]: Refusing update from remote peer 502ca883a5d74182a320d1d213c3d0c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250111 02:54:17.920696 7228 raft_consensus.cc:1270] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 3 FOLLOWER]: Refusing update from remote peer 502ca883a5d74182a320d1d213c3d0c0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20250111 02:54:17.921062 7636 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250111 02:54:17.921276 7664 consensus_queue.cc:1035] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250111 02:54:17.923247 7660 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.923791 7477 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.923904 7228 raft_consensus.cc:2949] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 5 to 6, VOTER ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } } }
I20250111 02:54:17.926460 7038 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet bd66331e08224e0a9b7b0db16d52e8aa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250111 02:54:17.927291 7050 catalog_manager.cc:5526] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 reported cstate change: config changed from index 5 to 6, VOTER ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3) evicted. New cstate: current_term: 3 leader_uuid: "502ca883a5d74182a320d1d213c3d0c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "502ca883a5d74182a320d1d213c3d0c0" member_type: VOTER last_known_addr { host: "127.0.103.1" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec1ece0a31324840be298ddc7e853d1b" member_type: VOTER last_known_addr { host: "127.0.103.2" port: 38955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf12e8128df5454699ffa9f896018b01" member_type: VOTER last_known_addr { host: "127.0.103.4" port: 36829 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250111 02:54:17.930644 7289 tablet_service.cc:1514] Processing DeleteTablet for tablet bd66331e08224e0a9b7b0db16d52e8aa with delete_type TABLET_DATA_TOMBSTONED (TS ddfa5c4f3c534b3b96b0269100068a4b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50908
I20250111 02:54:17.931325 7669 tablet_replica.cc:331] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: stopping tablet replica
I20250111 02:54:17.931524 7669 raft_consensus.cc:2238] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:17.931731 7669 raft_consensus.cc:2267] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:17.932422 7669 ts_tablet_manager.cc:1905] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250111 02:54:17.936385 7669 ts_tablet_manager.cc:1918] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20250111 02:54:17.936487 7669 log.cc:1198] T bd66331e08224e0a9b7b0db16d52e8aa P ddfa5c4f3c534b3b96b0269100068a4b: Deleting WAL directory at /tmp/dist-test-taskozm4WP/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1736563872491422-412-0/minicluster-data/ts-2-root/wals/bd66331e08224e0a9b7b0db16d52e8aa
I20250111 02:54:17.937094 7037 catalog_manager.cc:4872] TS ddfa5c4f3c534b3b96b0269100068a4b (127.0.103.3:37345): tablet bd66331e08224e0a9b7b0db16d52e8aa (table test-workload [id=65a3e92e9b2a4ae19fbb26f8568e3e9e]) successfully deleted
I20250111 02:54:19.909940 412 tablet_server.cc:178] TabletServer@127.0.103.1:37843 shutting down...
I20250111 02:54:19.921617 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:19.921896 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:19.922073 412 raft_consensus.cc:2238] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 3 LEADER]: Raft consensus shutting down.
I20250111 02:54:19.922348 412 raft_consensus.cc:2267] T bd66331e08224e0a9b7b0db16d52e8aa P 502ca883a5d74182a320d1d213c3d0c0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:19.922904 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:19.923025 412 raft_consensus.cc:2238] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:19.923151 412 raft_consensus.cc:2267] T 85776db200cd43268074d11d9c659f79 P 502ca883a5d74182a320d1d213c3d0c0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:19.923555 412 tablet_replica.cc:331] stopping tablet replica
I20250111 02:54:19.923660 412 raft_consensus.cc:2238] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:19.923823 412 raft_consensus.cc:2267] T 35ccdf9e9bdf46179da1c099ea14c3b6 P 502ca883a5d74182a320d1d213c3d0c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:19.938766 412 tablet_server.cc:195] TabletServer@127.0.103.1:37843 shutdown complete.
I20250111 02:54:19.953346 412 tablet_server.cc:178] TabletServer@127.0.103.2:0 shutting down...
I20250111 02:54:19.961541 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:19.961783 412 tablet_replica.cc:331] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b: stopping tablet replica
I20250111 02:54:19.961984 412 raft_consensus.cc:2238] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:19.962261 412 raft_consensus.cc:2267] T a0c1d0c792c4443c962a2a013e49e3af P ec1ece0a31324840be298ddc7e853d1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:19.962836 412 tablet_replica.cc:331] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b: stopping tablet replica
I20250111 02:54:19.962944 412 raft_consensus.cc:2238] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:19.963127 412 raft_consensus.cc:2267] T bd66331e08224e0a9b7b0db16d52e8aa P ec1ece0a31324840be298ddc7e853d1b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:19.963701 412 tablet_replica.cc:331] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b: stopping tablet replica
I20250111 02:54:19.963798 412 raft_consensus.cc:2238] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:19.963901 412 raft_consensus.cc:2267] T 85776db200cd43268074d11d9c659f79 P ec1ece0a31324840be298ddc7e853d1b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:19.977671 412 tablet_server.cc:195] TabletServer@127.0.103.2:0 shutdown complete.
I20250111 02:54:19.981865 412 tablet_server.cc:178] TabletServer@127.0.103.3:0 shutting down...
I20250111 02:54:19.989913 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:19.990173 412 tablet_replica.cc:331] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b: stopping tablet replica
I20250111 02:54:19.990355 412 raft_consensus.cc:2238] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 2 LEADER]: Raft consensus shutting down.
I20250111 02:54:19.990660 412 raft_consensus.cc:2267] T 85776db200cd43268074d11d9c659f79 P ddfa5c4f3c534b3b96b0269100068a4b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:19.991236 412 tablet_replica.cc:331] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b: stopping tablet replica
I20250111 02:54:19.991370 412 raft_consensus.cc:2238] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:19.991614 412 raft_consensus.cc:2267] T a0c1d0c792c4443c962a2a013e49e3af P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:19.992055 412 tablet_replica.cc:331] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b: stopping tablet replica
I20250111 02:54:19.992168 412 raft_consensus.cc:2238] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:19.992375 412 raft_consensus.cc:2267] T 35ccdf9e9bdf46179da1c099ea14c3b6 P ddfa5c4f3c534b3b96b0269100068a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:20.006938 412 tablet_server.cc:195] TabletServer@127.0.103.3:0 shutdown complete.
I20250111 02:54:20.011320 412 tablet_server.cc:178] TabletServer@127.0.103.4:0 shutting down...
I20250111 02:54:20.019950 412 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250111 02:54:20.020215 412 tablet_replica.cc:331] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01: stopping tablet replica
I20250111 02:54:20.020406 412 raft_consensus.cc:2238] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:20.020619 412 raft_consensus.cc:2267] T bd66331e08224e0a9b7b0db16d52e8aa P bf12e8128df5454699ffa9f896018b01 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:20.021176 412 tablet_replica.cc:331] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01: stopping tablet replica
I20250111 02:54:20.021286 412 raft_consensus.cc:2238] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:20.021456 412 raft_consensus.cc:2267] T 35ccdf9e9bdf46179da1c099ea14c3b6 P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:20.021919 412 tablet_replica.cc:331] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01: stopping tablet replica
I20250111 02:54:20.022017 412 raft_consensus.cc:2238] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250111 02:54:20.022181 412 raft_consensus.cc:2267] T a0c1d0c792c4443c962a2a013e49e3af P bf12e8128df5454699ffa9f896018b01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:20.035784 412 tablet_server.cc:195] TabletServer@127.0.103.4:0 shutdown complete.
I20250111 02:54:20.048303 412 master.cc:537] Master@127.0.103.62:43731 shutting down...
I20250111 02:54:20.055795 412 raft_consensus.cc:2238] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 1 LEADER]: Raft consensus shutting down.
I20250111 02:54:20.056008 412 raft_consensus.cc:2267] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250111 02:54:20.056087 412 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5c363f13bd8a4021b152ce6d3c94cc3d: stopping tablet replica
I20250111 02:54:20.069156 412 master.cc:559] Master@127.0.103.62:43731 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (8268 ms)
[----------] 14 tests from AutoRebalancerTest (187521 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (187521 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20250111 02:54:20.082052 412 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20250111 02:54:20.082123 412 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 7 messages since previous log ~5 seconds ago
I20250111 02:54:20.082155 412 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:4615: suppressed but not reported on 11 messages since previous log ~5 seconds ago
I20250111 02:54:20.082203 412 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:5204: suppressed but not reported on 9 messages since previous log ~30 seconds ago
I20250111 02:54:20.082242 412 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:643: suppressed but not reported on 8 messages since previous log ~173 seconds ago
I20250111 02:54:20.082305 412 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2367: suppressed but not reported on 46 messages since previous log ~18 seconds ago
I20250111 02:54:20.082341 412 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 30 messages since previous log ~8 seconds ago